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Abstract 

\ 

This  report  is  intended  to  serve  as  a resource  for  the 
development  of  management  and  instructional  guidelines  for 
computer-based  education  (CBE).  Although  the  data  in  it 
were  gathered  from  PLATO  projects  only,  they  represent  pro- 
jects which  varied  widely  in  target  populations  (elementary 
through  professional  students),  subject  matter  content,  type 
of  implementation,  and  size  and  scope.  Therefore,  it  is 
expected  that  the  report  will  be  useful  to  developer's  of  CBE 
in  gener  al and  not  bnly  to  PLATO  users.  ^ 

Critical  incidents  are  defined  in  terms  of  four  cri- 
teria. Then  ;more  than  125  case  histories  of  critical  inci- 
dents are  documented.  They  are  organized  by  topics,  rather 
than  projects,  which  in  effect  will  serve  as  a taxonomy  of 
matters  or  issues  which  are  critical  during  project  develop- 
ment.' The  report  also  includes  summaries  and  analyses  of 
the  processes  and  procedures  and  their  subsequent  effects. 

Just  a few  conclusions  will  be  given  here.  CBE  was 
implemented  most  smoothly  when  there  was  a conscious  effort 
to  develop  good  relations  with  instructors  and  administra- 
tors. Successful  projects  were  those  which  had  initial 
plans  for  such  matters  as  project  procedures,  organization, 
objectives  and  evaluation,.  Directors  who  planned  contin- 
gencies, in  case  of  failure  to  meet  goals,  increased  the 
probability  of  the  effectiveness  of  their  pro jects . A Evalua- 
tion was  essential,  throughout  lesson  development  as  well  as 
after  lesson  completion.  Many  models  of  staff  organization 
were  noted,  but  there  was  no  single  best  model  for  lesson 
development.  Successful  staff  members  were  those  whose 
expertise  was  not  limited  to  a single  area,  such  as  pro- 
gramming, but  who  had  breadth  in  many  areas  including 
teaching  in  an  interactive  environment. 
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Chapter  1 
iNTKUDUCTION 


A.  Background  and  Rationale 

PLATO  is  a relatively  new  technology  (cf.  Lyman,  1975), 
as  is  all  computer-based  education  (CBE).  When  PLATO  III, 
the  first  classroom-oriented  version  of  the  system,  was 
introduced,  the  major  educational  efforts  were  exploratory 
and  evolutionary.  Directors  of  these  early  projects  had  to 
formulate  plans  without  the  benefit  of  other  people's 
experience.  This  was  due  to  the  fact  that  CBE,  and  the 
PLATO  III  system  in  particular,  was  a novel  and  unique 
medium.  Previous  research  in  related  areas  such  as 
programmed  instruction  and  curriculum  development  was  an 
inadequate  source  of  information.  Project  directors  tended 
to  try  out  ideas  in  order  to  learn  from  experience. 
Sometimes,  they  shared  the  knowledge  gained  with  other 
projects.  For  the  most  part  this  information  was  not 
recorded . 

With  the  advent  of  the  PLATO  IV  system  a few  years 
later,  the  system's  capability  expanded  to  handling  hundreds 
of  terminals  simultaneously.  Directors  of  new  and  larger 
projects  had  to  make  decisions  not  only  about  instructional 
design,  but  also  about  project  management  and  organization. 
The  new  dimensions  of  CBE  made  it  necessary  to  revise  old 
concepts  and  sometimes  even  to  develop  new  procedures.  Di- 
rectors could  not  anticipate  all  aspects  of  planning  that 
would  be  necessary,  and  there  were  scant  resources  to  guide 
them.  Unlike  other  CBE  projects  (cf.  Faust,  197^)  each 
group  using  PLATO  made  its  own  decisions  about  organization, 
procedures,  staff  selection,  lesson  design,  and  evaluation. 
The  groups  varied  widely  in  size,  setting,  and  educational 
level.  As  knowledge  and  experience  accumulated,  staff 
members  of  each  group  made  modifications  or  even  complete 
revisions  of  one  or  more  aspects  of  their  work. 

Sometimes,  groups  who  were  working  under  different 
circumstances  came  to  similar  conclusions  about  effective 
ways  to  reorganize  or  to  improve  some  aspect  of  their 
project.  For  example,  two  projects  that  were  very  different 
in  size,  scope,  and  target  population  found  that  the  same 
organizational  structure  met  their  needs.  Sometimes,  how- 
ever, when  confronted  with  similar  decisions,  the  various 
groups  chose  different  alternatives  because  of  their  partic- 
ular circumstances.  For  example,  some  groups  decided  to 
adapt  available  curricular  materials  to  the  CBE  system, 
whereas  others  developed  new  curricula.  What  was  a good 
idea  for  one  project  was  not  necessarily  considered  to  be  a 
good  idea  for  another. 
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The  sum  total  of  this  experience  can  provide  a valuable 
data  base  for  Raiding  the  development  of  many  aspects  of 
future  CBE  projects.  Specific  documentation  of  circum- 
stances preceding  and  surrounding  these  incidents  and  deci- 
sions is  essential  so  that  the  consequences  may  be  under- 
stood in  proper  context.  This  kind  of  information  has  not 
been  generally  available  Although  most  projects  have 
provided  reports  and  summaries  of  their  activities,  these 
have  varied  considerably.  They  include  such  materials  as 
catalogs  of  lessons,  data  on  lesson  usage,  summaries  of 
student  performance,  and  comparison  to  control  groups.  ^'ost 
are  summative  evaluations,  giving  little  or  no  formative 
evaluation  data  (i.e.,  information  about  the  evolution  of 
project  processes  and  procedures)  . Those  reports  which  do 
include  some  of  this  information  tend  to  be  incomplete. 

They  are  prone  to  overlook  and  omit  decisions  and  events 
which  produced  negative  results.  In  some  situations,  where 
there  was  a complete  turnover  of  personnel,  the  report 
writers  tend  to  include  little  or  no  information  about  the 
earlier  phase  of  the  project. 

Outside  evaluators  have  written  about  some  aspects  of 
project  development,  but  these  are  limited  to  their  parti- 
cular focus.  House  (1973)  provided  extensive  process  docu- 
mentation about  one  implementation  effort  but  necessarily 
discussed  events  that  served  to  illuminate  the  author's 
topic,  the  politics  of  innovation  in  education.  Similarly, 
Mahler  and  his  colleagues  (1976)  documented  PLATO  courseware 
development,  but  the  information  is  general  and  lacks  the 
detail  that  would  be  needed  for  developing  guidelines  for 
the  future. 

B.  Purpose 

The  purpose  of  this  report  is  to  provide  a broad-hased 
set  of  case  histories  and  decisions  that  were  observed  to 
have  specific  effects  or  long-term  impact  on  the  smoothness 
of  pi^oject  implementation.  These  incidents  are  intended  to 
provide  a data  base  for  developing  guidelines  for  future  CPE 
projects  in  general  and  PLATO  projects  in  particular.  It  is 
also  intended  to  serve  as  a first  approximation  to  a taxon- 
omy of  factors  that  should  be  considered  in  project  manage- 
ment and  instructional  design. 

C.  Method 

1 . Resources 

Three  sources  of  information  were  used:  personal 
accounts  of  the  individuals  who  developed  lessons  and  were 
directly  involved  in  the  projects  or  were  in  a position  to 
make  personal  observations;  interviews  by  the  editor  with 
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some  directors  and  staff  members  of  various  projects;  and 
- published  documents  such  as  those  previously  listed. 

2.  Definitions  and  criteria  for  inclusion  of  cases 

Flanagan  (1954)  developed  a technique  for  arriving  at 
job  specifications  by  documenting  and  classifying  critical 
incidents.  He  defined  an  incident  as  critical  if  its  pur- 
pose was  clear  and  the  results  were  definite  enough  to 
"leave  little  doubt  about  its  effects."  This  definition  has 
been  adapted  here  for  evaluating  aspects  of  CBE  projects  and 
has  served  as  a starting  point  for  the  inclusion  of  inci- 
dents in  this  report.  Three  additional  definitions  were 
included  to  cover  other  kinds  of  situations.  Thus  there 
were  four  criteria  for  inclusion  of  incidents. 

1)  A decision  or  incident  was  considered  critical 
if  the  circumstances  surrounding  it  were  clear 
and  the  results  or  effects  were  definite. 

2)  There  were  effects  or  results  of  situations  that 
could  be  traced  to  the  fact  that  a decision  was 
not  made.  Therefore,  for  some  incidents  included 
here  the  effects  were  traced  to  the  failure  of  a 
person  in  a position  of  responsibility  to  make 
a particular  decision. 

3)  There  were  situations  in  which  a decision  had  to 
be  made  to  choose  between  alternatives.  The 
observed  or  reported  advantages  and  disadvantages 
of  the  alternatives  provided  valuable  information. 

4)  In  some  instances,  a successful  scheme  or  decision 
evolved  through  repeated  iterations  of  situation- 
decision-effect  cycles.  Since  these  situations  met 
the  ultimate  objective  of  this  report,  they  were 
included . 

3.  Procedures 

The  initial  thrust  was  an  informal  "brain  storming" 
session  in  which  the  authors  reported  incidents  which  came 
to  mind.  As  each  individual  recalled  events  about  partic- 
ular aspects  of  project  development,  the  memory  of  others 
was  stimulated  and  activated.  This  initial  set  of  events 
was  then  organized  into  what  seemed  like  a reasonable  set  of 
topic  headings.  Subsequently  the  authors  held  meetings  to 
add  more  information  and  help  clarify  the  definitions  and 
criteria  for  including  incidents  in  this  report.  The  editor 
interviewed  12  additional  CBE  staff  members  representing  9 
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different  projects,  and  reviewed  more  than  20  project  papers 
and  reports  for  appropriate  information. 

In  most  projects,  several  critical  incidents  resulted, 
often  in  various  aspects  of  instruction  and  management. 

Since  the  cases  cited  below  were  classified  according  to 
topics,  rather  than  projects,  it  was  necessary  to  repeat  the 
circumstances  in  some  cases  in  order  to  make  the  report  more 
readable . 

4.  Credibility 

It  should  be  noted  that  the  incidents  reported  below 
may  be  somewhat  biased.  Memory  tends  to  be  selective; 
people  tend  to  remember  the  dramatic  and  perhaps  fail  to 
recall  the  ordinary.  They  may  be  inclined  to  recall  nega- 
tive incidents  more  frequently  than  positive.  In  fact, 
recall  may  sometimes  be  somewhat  inaccurate.  We  have  tried 
to  minimize  this  problem  by  obtaining  reports  of  an  incident 
from  more  than  one  source  where  possible. 

The  role  of  an  individual  within  a project  also  preju- 
dices the  incidents  he  recalls  (cf.  Flanagan,  1954).  In 
order  to  keep  this  bias  to  a minimum,  the  people  selected  to 
be  interviewed  were  chosen  to  represent  different  levels  of 
responsibility  and  duties.  The  authors  of  this  report  also 
represent  many  projects  and  roles  within  them.  It  is  there- 
fore hoped  that  this  kind  of  selection  bias  is  at  a minimum. 

The  experiences  reported  here  are  not  intended  to  be 
used  as  a final  authority  for  judging  the  adequacy  of  CBE 
projects.  In  some  instances  the  evidence  is  scanty  and 
inadequate  for  making  generalizations.  The  intent  is  to 
provide  tentative  and  preliminary  information  that  can  be 
useful  as  a guide  for  the  planning  and  management  of  all 
facets  of  new  projects. 

Some  of  the  "lessons  learned"  will  appear  to  be  no  more 
than  common  sense.  But  what  is  obvious  in  hindsight  was  not 
so  clear  or  even  visible  in  foresight.  The  purpose  of 
reporting  failures,  in  addition  to  successes,  is  not  to 
criticize  but  rather  to  emphasize  the  impact  of  the  various 
factors  in  establishing  and  managing  a CBE  site. 
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Chapter  2 

PROJECT  ESTABLISHMENT 


A.  Introducing  CBE 

Introducing  CBE  involved  special  problems,  above  and 
beyond  those  usually  connected  with  instructional  innova- 
tion. The  new  medium,  the  computer,  was  still  a relatively 
recent  invention.  Most  people  had  not  worked  directly  with 
computers  and  were  not  yet  comfortable  with  them.  Some  felt 
threatened  by  them  or  were  afraid  of  being  replaced.  Others 
expected  the  new  medium  to  be  a great  cure-all  for  many  of 
their  problems.  Another  unusual  circumstance  was  that 
lessons  were  often  used  for  instruction  as  soon  as  they  were 
finished  so  that  instructors  were  unable  to  familiarize 
themselves  with  the  lesson  content  and  format  before  their 
students  began  to  use  them  (unless  the  instructors  were 
themselves  the  authors  of  the  lessons)  . 

The  way  in  which  the  CBE  system  was  introduced  at  an 
institution  affected  staff  as  well  as  students.  The  extent 
and  the  nature  of  public  relations  and  orientation  had  a 
decided  effect  on  the  acceptance,  cooperation,  and  expecta- 
tions of  those  associated  with  the  project.  The  incidents 
cited  below  show  three  factors  that  engendered  good  will 
toward  CBE  and  instructor  acceptance  and  cooperation. 

Broadly  categorized,  they  are:  (1)  involvement  (2)  famil- 
iarity with  CBE  and/or  the  CBE  lessons  and  (3)  volunteers  as 
instructors.  Desirable  attitudes  resulted  when  a deliberate 
effort  was  made  to  provide  enough  time  for  the  instructor  to 
work  on  the  CBE  system  or  to  take  some  meaningful  part  in 
the  p>'’ect,  like  critiquing  lessons  or  assuming  part  of  the 
compu'L  . responsibility  for  instructional  management. 
Conver:.c.  , other  cases  indicate  that  when  teachers  were  not 
adequately  familiar  with  the  CBE  system  or  were  insuffi- 
ciently involved,  they  resisted  using  the  CBE  lessons  and 
did  not  make  an  effort  to  integrate  them  into  their  regular 
classroom  instruction. 

Expectations  could  not  always  be  met  as  anticipated. 

In  one  such  instance  the  project  director  planned  some 
alternatives  in  project  procedures  which  enabled  the  project 
to  progress  even  though  fewer  terminals  were  available  than 
had  been  anticipated.  In  other  instances,  failure  of  the 
director  to  make  contingency  plans  or  to  furnish  explana- 
tions for  unfulfilled  promises  led  to  misunderstandings  and 
bad  feelings. 
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1 . Instructor  acceptance  and  cooperation 

Case  2A1.1  - Volunteers  as  cooperative  instructors 

CBE  terminals  were  to  be  placed  in  a prespecified 
number  of  classes  within  a large  institution.  The  project 
director  had  to  decide  on  a plan  for  selecting  the  limited 
number  of  instructors  who  would  have  terminals  in  their 
classrooms.  He  decided  to  ask  for  volunteers.  The  instruc- 
tors proved  to  be  cooperative  throughout  the  project  in 
spite  of  errors  in  lessons  or  other  problems  that  arose. 
Cooperation  was  good  even  among  those  teachers  who  knew  that 
terminals  would  not  be  available  to  them  the  following  year. 

Case  2A1.2  - Developing  good  will  among  instructors 

At  one  site,  as  an  experiment,  an  entire  course  was 
being  rewritten  to  become  self-paced.  (Parallel  CBE  and 
non-CBE  portions  of  the  course  were  to  be  developed  simul- 
taneously.) It  was  an  intricate  patchwork  of  materials  from 
many  media.  The  CBE  project  director  had  taught  this  course 
and  presumbably  knew  the  course  instructors.  Early  in  the 
project,  he  decided  that  he  and  his  staff  should  work  very 
hard  to  establish  good  relationships  with  the  instructors 
and  developers  of  non-CBE  materials  in  order  to  keep  a 
natural  animosity  from  developing.  He  lent  the  other  group 
staff,  gave  them  all  signons,  taught  them  to  play  CBE  games, 
reviewed  lessons  between  project  and  traditional  staff,  and 
paid  what  any  unknowing  outsider  would  deem  "unnecessary" 
attention  to  details  that  reduced  jealousy  between  groups. 

He  even  went  so  far  as  to  halt  lesson  polishing  when  CBE 
lessons  reached  a level  where  they  were  clearly  superior  to 
the  conventional  materials.  To  have  polished  them  more 
would  not  only  have  needlessly  added  to  development  time  but 
also  might  have  fostered  feelings  of  jealousy  in  the  devel- 
opers of  the  materials  using  conventional  media.  The  result 
was  acceptance  by  the  conventional  staff,  and  relatively 
smooth  implementation. 

Case  2A1.3  - Giving  instructors  responsibilities 

In  one  large  curriculum  project  the  director  decided  to 
put  curriculum  management  under  computer  control.  That  is, 
the  computer  kept  a complete  record  for  each  student  and  on 
that  basis  decided  which  lesson  to  present  next.  The 
instructors  were  unfamiliar  with  the  lessons  and  with  the 
hardware.  Although  they  had  the  option  of  over-iding  the 
computer  and  setting  up  their  own  sequence,  the>  rarely 
exercised  it.  They  viewed  the  CBE  materials  as  a separate 

rt  of  the  regular  curriculum. 
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Epilog.  In  subsequent  years,  the  decision  was  made  to 
abandon  computer  management  and  to  give  that  responsibility 
to  the  instructors.  By  that  time  they  were  more  familiar 
with  both  the  lessons  and  the  system.  They  had  to  do  much 
more  work,  diagnosing  and  prescribing  lessons  for  each 
student.  However,  it  gave  them  a sense  of  power  and  control 
over  the  machine,  and  they  had  more  students  do  more  lessons 
than  in  previous  years.  The  students,  in  turn,  were  more 
interested  because  the  lessons  were  more  appropriate  and  the 
teacher  showed  a vital  interest  and  participated  actively. 

Case  2A1.4  - Asking  instructors  to  critique  lessons 
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One  group  of  authors  wanted  to  evaluate  their  materials 
with  students  from  the  target  population.  They  decided  that 
the  instructors  would  be  more  inclined  to  cooperate  if  they 
were  involved  in  a meaningful  way,  so  they  asked  the 
instructors  to  review  and  critique  the  lessons.  At  that 
time  the  lessons  were  still  in  a primitive  state  and  had  not 
been  carefully  pretested.  Nevertheless,  three  of  the 
instructors  were  positive  in  their  reviews  and  used  CBE 
lessons  with  one  or  more  of  their  classes.  One  instructor 
wrote  a negative  critique  and  decided  not  to  use  CBE  at  all. 

2.  Instructor  resistance 

Case  2A2.1  - Insufficient  "hands  on"  experience 

In  the  first  phase  of  the  development  of  a large 
curriculum  project,  lessons  were  developed  by  the  CBE  staff 
who  were  not  part  of  the  institution  that  would  use  them. 

The  decision  about  the  educational  level  at  which  to  write 
was  made  exclusively  by  the  CBE  staff.  They  also  decided  on 
lesson  content.  Instructors  who  would  use  these  materials 
were  solicited  for  some  advice  about  lesson  content  and 
general  demonstrations  were  given,  but  no  provision  was  made 
for  "hands  on"  experience  with  the  materials.  This  was 
partly  due  to  the  fact  there  was  neither  lead  time  nor  a 
backlog  of  lessons;  lessons  were  written  and  used  immedi- 
ately. Also  systems  limitations  made  it  difficult  to  obtain 
time  for  instructors  to  try  out  the  lessons.  One  further 
obstacle  was  that  teachers  had  to  go  to  a special  room  to 
use  the  terminals.  As  a result,  most  of  the  instructors 
treated  the  CBE  experience  as  a supplementary  activity  and 
not  as  an  integrated  or  important  part  of  the  curriculum. 
They  wanted  to  have  little  to  do  with  it  and  resisted 
becoming  involved. 


Epilog . When  a special  time  was  later  set  aside  for  them  to 
come  to  try  out  the  materials,  only  1 out  of  25  instructors 
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showed  up  even  though  time  for  perusal  was  scheduled  for 
their  convenience. 

Case  2A2.2  - Insufficient  familiarity  with  lessons 

Lessons  in  a given  subject  were  developed  and  validated 
with  a particular  subject  population  and  were  shown  to  be 
effective.  The  lessons  were  then  offered  to  another  insti- 
tution which  had  a similar  student  body.  The  institution 
did  arrange  for  review  time  for  any  interested  instructor. 
The  department  chairman  (who  was  very  positive  toward  the 
material)  scheduled  enough  time  so  that  members  of  every 
class  could  complete  all  appropriate  materials.  However 
because  instructors  and  authors  of  the  materials  were  busy, 
no  formal  attempt  was  made  to  insure  that  instructors  at  the 
new  site  were  aware  of  the  detailed  content  of  the 
materials. 

Very  few  of  the  instructors  took  the  time  to  examine 
the  materials.  Virtually  no  attempt  was  made  to  coordinate 
class  instruction  with  the  materials.  Most  instructors 
simply  sent  their  students  to  the  CRE  terminals  at  the 
appointed  time  as  an  independent  activity.  students  often 
got  instruction  via  the  CBE  system  for  which  they  had  no 
class  preparation,  or  instruction  that  duplicated  earlier 
class  presentations.  Student  attitudes  were  extremely  hos- 
tile since  they  saw  the  PLATO  lessons  as  a waste  of  their 
time.  Instructor  attitudes  were  only  slightly  better. 

Epilog . Following  the  above  experience,  an  effort  was  made 
to  involve  instructors  in  the  revision  of  the  original 
materials.  Although  only  minor  revisions  were  in  fact 
carried  out,  most  of  the  instructors  viewed  all  materials 
and  participated  in  making  up  a written  guide  for  the 
coordination  of  the  materials  with  classroom  activities. 
Subsequent  student  and  instructor  attitudes  toward  CRE  and 
the  materials  were  above  average  for  the  institution, 
although  still  lower  than  those  at  the  site  that  first 
developed  the  materials. 

Case  2A2.3  - Insufficient  familiarity  with  CRE 

Lessons  were  developed  at  a curriculum  center  for  use 
at  another  institution.  Since  lessons  were  used  by  students 
just  as  soon  as  they  were  completed,  instructors  did  not 
have  a chance  to  see  them  first,  and  were  frequently  too 
busy  to  make  time  to  view  them  at  all.  They  were  uncomfort- 
able with  the  CBF  system  and  unfamiliar  with  the  lesson 
content . 
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Epilog.  Before  the  second  year  began,  the  project  director 
made  time  available  for  the  instructors  to  have  "hands  on" 
experience  before  classes  started.  They  felt  more  at  ease 
with  the  lessons  and  this  attitude  was  reflected  in  the 
students.  Instructors  also  were  able  to  help  students  who 
were  having  difficulties. 

Case  2A2.4  - Schism  between  staffs 

At  one  site  the  staff  felt  that  the  entire  curriculum 
needed  to  be  made  more  relevant  for  the  students  and  decided 
it  should  be  revised.  At  about  the  same  time  funds  were 
made  available  to  get  some  CBE  terminals.  Without  further 
consulting  the  course  instructors,  the  course  director 
initiated  a project  to  develop  an  entirely  new  curriculum 
while  simultaneously  initiating  the  first  use  of  CBE  at  the 
institution . 

He  hired  12  staff  members  to  write  the  new  curriculum, 
10  of  them  new  people.  The  plan  was  to  have  the  "tradi- 
tional" staff  adapt  the  content  of  their  courses  to  the 
needs  of  the  CBE  research  project.  They  were  expected  to 
isolate  all  of  their  lectures  that  dealt  with  a specified 
topic;  then  the  content  was  to  be  given  to  the  CBE  staff  for 
lesson  writing.  The  traditional  instructors  were  expected 
to  teach  the  selected  topics  at  a particular  time  so  that 
while  some  of  the  students  attended  the  lectures,  others 
simultaneously  learned  the  same  material  via  CBE.  This 
entailed  a considerable  amount  of  work  for  traditional 
course  instructors  and  they  did  not  perceive  CBE  as  a 
benefit  either  to  the  course  or  themselves.  Furthermore 
they  saw  the  CBE  project  as  research  that  would  never  be 
implemented  so  they  could  just  wait  it  out.  They  had  no 
motivation  to  cooperate  and  were  slow  in  turning  over  lesson 
materials  fc"  the  CBE  authors  to  use.  They  did  not  come  to 
the  terminals,  either  to  try  the  lessons,  to  observe  the 
students  or  even  just  to  play  on-line  games. 

A further  source  of  friction  was  the  fact  that  the 
traditional  staff  had  received  preferential  treatment  over 
the  other  staff  at  the  school.  Because  the  traditional 
staff  members  were  developing  original  study  programs,  they 
were  exempt  from  some  of  the  duties  that  instructors  in 
other  parts  of  the  school  were  expected  to  perform.  They 
were  somewhat  ego  involved  and  maintained  a "hands  off" 
policy  toward  their  lessons.  They  were  hardworking,  but 
often  considered  arrogant  by  other  instructors.  As  a 
result,  they  were  disinterested  and  professionally  conde- 
scending. No  one  served  as  an  interface  between  the  two 
staffs.  To  further  complicate  the  matter,  channels  of 
communication  were  inadequate  between  the  leaders  of  the  two 
groups  and  between  leaders  and  their  staffs. 
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Epilog . At  about  the  time  this  project  was  ending,  the  CBE 
administrator  approached  staff  in  other  courses  and  offered 
them  use  of  the  CBE  facilities.  In  this  instance,  in  con- 
trast to  the  earlier  one,  CPE  was  offered  as  a means  of 
meeting  the  instructors'  perceived  needs.  They  were  moti- 
vated to  use  the  CBE  system  to  solve  their  training 
problems . 

3.  Expectations  and  consequent  misunderstandings 
Case  2A3.1  - Who  can  use  terminals 

Staff  members  of  one  organization  were  notified  that 
they  were  to  receive  a classroom  of  terminals.  They  were  not 
sure  that  they  could  fully  utilize  the  CBE  terminals. 

Rather  than  waiting  until  the  uncertainty  was  resolved,  they 
spread  the  word  that  the  terminals  would  be  coming  and 
invited  suggestions  about  their  use  in  other  departments. 
Later  they  found  that  there  was  no  excess  of  terminal  avail- 
ability, so  they  reneged  and  restricted  usage.  The  result 
was  antagonism  throughout  the  organization,  and  members  of 
other  departments  were  reluctant  to  act  as  lesson  reviewers 
or  to  provide  students  for  lesson  trials. 

Case  2A3.2  - Fewer  terminals  than  promised 

At  one  site,  a given  number  of  terminals  were  promised. 
Due  to  a manufacturer's  delay,  fewer  were  delivered  ini- 
tially than  anticipated.  Conseqently  the  ratio  of  authors 
to  terminals  was  about  3 to  1 . Since  the  PLATO  terminals 
were  available  22  hours  a day,  the  staff  decided  to  resolve 
the  problem  by  working  in  three  split  shifts.  Higher  pay 
was  provided  as  an  incentive  for  night  work.  The  authors 
claimed  that  this  system  reduced  tensions.  But  the  admini- 
strators said  that  extra  effort  was  needed  to  coordinate 
staff.  They  reported  some  problems  since  the  staff  members 
were  not  always  under  supervision,  and  production  standards 
were  not  well  defined.  Moreover,  the  split  shifts  appar- 
ently fostered  cliquishness  and  division  among  the  authors. 

Case  2A3.3  - Hot  as  reliable  as  anticipated 

One  institution  decided  to  get  terminals  because  vali- 
dated curriculum  material  was  already  available  in  an  area 
they  really  needed.  Instructors  were  disappointed,  however, 
because  the  system  was  not  as  reliable  as  had  been  antici- 
pated. Administrators  decided  they  had  been  "used"  by  the 
developers  for  testing  hardware.  The  developers  sent 
personnel  to  help  solve  problems,  and  in-service  author 
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training  for  instructors  was  promised.  This,  coupled  with 
the  fact  that  extra  funding  would  be  available  if  the  CBE 
' system  was  used,  led  to  an  improved  attitude  on  the  part  of 

^ the  administrators  as  well  as  the  teaching  staff. 

t 

Case  2A3.4  - For  whom  CBE  is  useful 

The  president  of  one  institution  saw  use  of  the  CBE 
system  as  a chance  to  get  outside  funding  and  to  reduce 
dependency  on  tenured  teachers,  many  of  whom  belonged  to 
unions.  Therefore,  he  decided  to  assign  non-tenured 
teachers  the  task  of  becoming  CBE  authors.  It  was  also  the 
case  that  these  individuals  were  in  charge  of  remediation 
for  open-admissions  students.  As  a result,  instruction  via 
' CBE  was  perceived  by  tenured  staff  as  a cheap  treatment  for 

' the  poorer  students  and  as  a means  of  reducing  the  power  of 

the  union  by  cutting  back  on  the  need  for  tenured  teachers. 
They  also  suspected  that  the  CBE  system  was  used  only  for 
minorities  and  not  at  prestigious  universities.  Persons 
assigned  to  aid  in  establishing  the  site  were  unfamiliar 
with  CBE  and  missed  the  opportunity  to  correct  these  impres- 
sions. (CBE  was,  in  fact,  used  by  prestigious  universities, 
and  this  particular  imple.mentation  was  an  attempt  to  bring 
advanced  technology  to  schools  with  fewer  resources.)  But 
because  of  this  false  impression  and  the  policy  of  using  CBE 
solely  for  remediation,  senior  staff  and  members  of  prestige 
departments  avoided  using  CBE.  Students  assigned  to  use  CBE 
became  identified  as  "slow"  or  "dumb". 

B.  Choice  of  Project  Director 

PLATO  projects  varied  widely  in  goals,  scope,  and  rate 
of  development.  Consequently  directors  were  chosen  under 
different  circumstances.  Some  were  individual  professors 
who  initiated  and  developed  the  project  on  their  own  initia- 
tive. They  learned  the  TUTOR  programming  language,  authored 
lessons  and  generally  explored  the  potential  of  the  PLATO 
system.  After  they  gained  experience  in  many  facets  of 
lesson  development,  they  expanded  their  staff  but  continued 
to  be  actively  involved.  Other  projects  were  intended  to  be 
demonstrations  of  use  of  the  PLATO  system.  In  some 
instances  the  director  was  chosen  from  among  the  instructor- 
authors  who  would  use  the  materials;  in  other  instances  an 
overall  project  director  was  appointed  to  manage  diverse 
projects  within  a given  institution.  Such  a director  was 
not  always  available  at  the  time  a PLATO  project  was  esta- 
blished. As  might  be  expected,  overall  lesson  production 
was  impeded  and  slowed  in  projects  which  were  initiated 
without  a leader  or  with  a temporary  one. 
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Even  when  such  a director  was  available,  he  sometimes 
lacked  the  essential  expertise.  CBF  was  so  new  that  there 
was  no  pool  of  available  leadership  in  the  field  per  se . 
Consequently  the  new  directors  were  experts  in  perhaps  one 
relevant  field  such  as  subject  content,  curriculum  develop- 
ment, computer  programing,  or  administration.  As  it  turned 
out,  the  most  successful  leaders  were  those  who  were  content 
experts  or  curriculum  developers  and  who  took  the  time  to 
learn  about  instructional  design  and  all  about  PI.ATO. 

In  some  instances  the  director  expected,  or  was 
expected  to  carry  on  some  other  responsibilites  in  addition 
to  managing  the  project.  Such  individuals  did  not  have  time 
to  become  adequately  knowledgeable  about  the  workings  of  the 
PLATO  system.  Not  only  did  they  not  have  first-hand  PLATO 
experience  themselves,  they  did  not  spend  enough  time 
observing  the  production  process.  Therefore,  when  manage- 
ment and  policy  decisions  had  to  be  made,  they  were  neces- 
sarily based  on  second-hand,  or  sometimes  conflicting 
reports.  Many  of  these  decisions  were  ill-advised  and 
resulted  in  staff  dissension  and  fragmentation.  One  direc- 
tor appointed  a coordinator  to  alleviate  the  problem.  It 
was  not  successful  because  the  coordinator  was  not  given  the 
authority  he  needed  to  carry  out  his  responsibilities. 

1 . Availability 

Case  2B1.1  - Leaderless  project 

A new  CBE  project  was  established  but  a director  could 
not  be  found.  The  individual  who  was  responsible  for 
funding,  but  not  for  project  outcomes,  decided  that  it  would 
be  better  to  get  something  going  rather  than  wait  for  a 
qualified  leader.  Consequently  he  hired  staff  members,  some 
of  them  rejects  from  other  projects,  and  allowed  them  to 
work  autonomously.  Each  person  developed  lessons  indepen- 
dently. One  member  of  the  group  expected  to  be  appointed 
leader  eventually.  When  a project  director  finally  did 
arrive,  several  months  later,  his  efforts  to  organize  and 
implement  a work  plan  were  thwarted  by  individual  members 
who  had  vested  interests  in  materials  already  developed  and 
in  directions  already  taken.  Productivity  of  the  group  fell 
and  much  time  was  spent  in  behind-the-scenes  struggles  for 
power . 

Case  2E1.2  - Temporary  administrator 

A project  was  tasked  with  implementing  an  experimental 
curriculum  requiring  a variety  of  high-level  cognitive  and 
affective  skills.  The  project  was  expected  to  last  for  three 
years  including  training  of  staff,  development  of  CPE 
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materials  and  evaluation  of  their  cost  and  instructional 
effectiveness.  It  was  difficult  to  find  a suitable  indivi- 
dual to  oversee  the  curriculum  development  since  highly 
technical  expertise  as  well  as  curriculum  development  skills 
were  needed.  The  curriculum  developer  joined  the  project 
six  months  after  its  beginning.  Meanwhile  a temporary 
leader  was  in  charge.  Fie  was  faced  with  two  alternatives. 
Although  not  qualified,  he  could  make  fundamental  decisions 
for  the  project  which  would  allow  it  to  begin  productive 
work  from  the  beginning.  Alternatively,  he  could  postpone 
these  decisions  until  the  arrival  of  the  curriculum  devel- 
oper and  allow  the  project's  authors  to  develop  lessons  in 
whatever  areas  they  felt  might  be  useful  to  the  project.  He 
selected  the  second  alternative. 

The  lessons  which  were  developed  in  the  six  month 
period  before  the  curriculum  developer  arrived  were  for  the 
most  part  unuseable;  they  simply  did  not  fit  into  his  plan 
for  the  project.  Ultimately  the  project  failed  to  meet  its 
intermediate  and  final  production  deadlines.  The  inability 
to  use  all  of  the  project's  time  effectively  was  probably 
detrimental  to  its  successful  conclusion.  Moreover,  the 
early  loss  of  time  forced  a hasty  and  slipshod  manner  of 
lesson  development  in  attempts  to  meet  the  deadlines. 

2.  Diverse  qualifications  of  directors 

Case  2B2.1  - Content  expert  studied  CDf^ 

In  at  least  four  projects  the  director  was  a content 
expert  who  had  teaching  experience  and/or  had  taken  courses 
in  instructional  design  but  had  no  experience  with  CRF!. 
Rather  than  begin  staffing  and  rely  on  experience  with  other 
media  for  setting  project  policies  and  goals,  the  new 
director  spent  one  or  more  years  learning  the  TUTOR  language 
sufficiently  well  to  produce  and  test  several  alternative 
types  of  instructional  approaches  that  might  make  full  use 
of  the  medium. 

Therefore  before  he  began  full  staffing  he  had  a good 
understanding  of  the  alternatives  in  project  development  and 
was  able  formulate  clear  goals.  The  projects  ultimately 
produced  large  amounts  of  good  material  on  a tight  schedule. 
This  was  so,  even  though  problems  sometimes  arose  in  the 
selection  of  an  experienced  programming  staff  whose  views 
fit  in  with  those  of  the  director. 

Case  ?n?.2  - Expertise  in  instructional  development 

The  individual  who  was  chosen  to  head  one  project  had 
experience  developing  instructional  materials  for  the 
audience  and  institution  which  was  implementing  CBR.  The 
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staff  he  chose  had  varied  backgrounds.  Some  were  experi- 
enced teachers  and  others  were  subject  matter  experts.  Only 
one  other  person  had  experience  developing  instructional 
‘ materials.  The  lessons  produced  under  his  direction  were 

for  the  most  part  effective  and  useable.  The  exoerience  of 
this  director  in  the  production  of  instructional  materials 
seemed  to  m.ake  him  aware  of  the  need  to  make  maximum  use  of 
all  available  time.  When  delivery  of  terminals  for  his  site 
was  delayed,  he  used  the  time  to  teach  the  basics  of 
j instructional  design  to  his  staff  and  planned  for  sharing 

, terminals  with  a nearby  site. 

Case  2B2.3  - Non-adaptive  application  of  expertise 

i 

, The  project  leader  was  experienced  in  innovative  cur- 

riculum development,  particularly  for  special  subgroups  of 
students.  He  decided  to  transfer  the  old  materials  to  CBf, 
using  his  same  lessons  and  instructional  strategies.  He  did 
not  take  into  account  the  differences  between  CPE  and  class- 
room interactions.  Many  of  the  lessons  were  not  suitable 
and  students  rebelled  at  doing  them.  Many  lessons  had  to  be 
scrapped  or  essentially  rewritten.  Considerable  time  was 
wasted  and  the  project  fell  behind  its  goals. 

Case  2B2.4  - Superb  program.mer 

An  administrator  appointed  a project  head  who  was 
experienced  in  a subject  area  related  to  that  intended  for 
CBE,  but  had  no  teaching  experience  with  the  target 
population  nor  with  the  exact  subject  matter.  He  was 
apparently  a hotshot  in  programming.  The  result  was  that  he 
directed  the  efforts  of  the  project  to  developing  an  exten- 
sive computer-managed  instruction  capability.  Insufficient 
time  was  allocated  to  writing  instructional  material.  There 
was  not  enough  time  to  produce  as  many  lessons  as  were 
needed,  and  very  little  time  was  allowed  for  student  trials. 
Many  of  the  lessons  that  were  produced  turned  out  to  be 
unsuitable  for  the  intended  students. 

Case  2B2.5  - Expertise  in  administration 

In  several  instances,  the  project  superv  sors  were 
administrators  who  had  neither  experience  teaching  the 
target  population  nor  subject  matter  expertise.  They  made 
the  decision  to  discard  certain  lessons,  based  on  their 
reactions  that  the  lessons  were  too  "boring"  or  "too  easy." 
But  the  authors  felt  that  the  presentation  was  at  an  appro- 
priate difficulty  level  for  the  students.  They  convinced 
the  directors  to  allow  the  lessons  to  run  and  to  let  the 
data  be  used  as  a basis  for  decision.  Data  showed  that 
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students  liked  the  lessons  and  also  had  a favorable 
attitude.  The  lessons  were  retained  and  used. 

3-  Part-time  leaders  of  large  projects 

Case  2D3.1  - Too  many  duties 

An  individual  was  hired  to  be  general  supervisor  of 
several  CBE  projects  and  to  be  directly  in  charge  of  one 
of  them.  In  addition,  he  brought  with  him  from  his  previous 
position  some  continuing  projects.  He  decided  that  he  needed 
time  to  learn  to  write  lessons  to  get  a better  feeling  for 
what  was  involved.  However  he  could  not  find  time  to  do  so, 
nor  could  he  even  find  tine  to  observe  students  working  at 
the  terminals.  In  many  cases  he  made  ill-advised  decisions 
and  judgments  about  what  kinds  of  lessons  would  be  success- 
ful. Staff  members  became  increasingly  reluctant  to  ask  for 
his  opinion  on  such  things  and  worked  independently.  Even- 
tually he  had  almost  no  time  to  devote  to  the  CBE  project. 
The  staff  splintered  into  factions  and  morale  was  low. 

Case  2B3.2  - Coordinator,  responsibility  without  authority 

In  four  different  projects,  the  leader  of  the  curri- 
culum group  was  also  responsible  for  many  other  activities 
and  had  no  time  to  engage  in  day-to-day  operation  of  the 
group.  He  made  policy  decisions  which  were,  in  most  cases, 
made  solely  on  the  basis  of  reports  rather  than  personal 
observation  of  the  situation  or  the  materials  being  devel- 
oped. He  appointed  a group  coordinator  to  give  day-to-day 
supervision.  The  coordinator  delegated  assignments  but  was 
given  no  authority  to  make  and  enforce  decisions.  The 
leader  gave  no  general  guidelines  and  left  planning  to  the 
coordinator.  The  leader  encouraged  group  members  to  report 
problems  directly  to  him  rather  than  work  though  the  coor- 
dinator. Decisions  resulting  from  such  conferences  were 
often  revealed  to  the  group  before  the  coordinator  learned 
of  them  or  had  a chance  to  give  counter  arguments.  The 
coordinator  eventually  became  little  more  than  an  assistant 
to  the  project  leader.  Daily  operation  of  the  group  became 
fractionated  since  the  group  members  would  only  follow 
directions  of  the  overall  leader  who  was  rarely  present  or 
available.  A good  deal  of  group  effort  was  spent  in 
"political"  infighting.  There  was  no  overall  structure  or 
framework  for  lesson  development.  The  totality  of  lessons 
was  not  a coordinated  whole,  but  fragmented  pieces. 

In  some  cases  the  director  received  conflicting 
information.  Since  he  did  not  have  time  to  verify  stories, 
he  resolved  the  conflicts  by  counting  the  number  of  people 
telling  the  same  story.  He  then  made  decisions  about 
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project  policies  on  this  basis.  Sometimes  this  resulted  in 
the  over-ruling  of  unpopular  suggestions  that  had  been  made 
, by  an  experienced  instructional  designer  who  was  nom.inally 

responsible  for  day-to-day  operation  of  the  group.  The 
group  went  leaderless,  morale  fell,  and  productivity  was 
’ almost  nil. 

Epilog.  The  entire  group  was  reorganized  under  a new 
t director.  Although  the  newdirector  could  only  devote  a 

fraction  of  his  time  to  administering  the  project,  members 
j of  the  groups  had  gained  enough  experience  so  that  they  were 

< able  to  work  independently.  Even  so,  deadlines  were  so  near 

by  the  time  the  groups  became  productive  that  only  minor 
. parts  of  the  original  production  goals  were  reached  and 

i overall  quality  of  the  materials  was  mediocre  at  best. 

C.  iltaff  Selection 

The  effectiveness  of  each  project  was  sometimes  impeded 
and  sometimes  enhanced  by  the  way  in  which  staff  were 
selected  and  trained. 

In  order  to  select  a project  staff,  the  project  leader 
needed  to  specify  the  qualifications  he  was  looking  for. 
Early  in  the  development  of  PLATO  projects  it  was  known  that 
successful  lesson  authors  were  experienced  in  more  than  one 
area,  such  as  lesson  design,  subject  matter,  and 
programming.  However  there  we^'e  ''ery  few  people  around  with 
all  of  these  qualifications.  In  addition  there  were  no 
formal  training  programs  in  the  early  years,  so  individuals 
learned  on  their  own  or  project  directors  developed  their 
own  training  in  whatever  way  they  could.  The  first  formal 
author  training  program  underwent  several  iterations  before 
it  was  evaluated  as  successful  three  years  after  PLATO  IV 
was  implemented  (Francis,  1975). 

Directors  were  not  always  free  to  choose  the  entire 
staff.  Sometimes  individuals  were  carried  over  from  a 
previous  project.  Sometimes  new  staff  were  hired  simply 
because  they  were  available  and  interested  at  a propitious 
time.  Some  staff  were  students  who  wanted  to  learn  how  to 
author  PLATO  lessons.  Others  were  individuals  with  exper- 
tise in  perhaps  just  one  area  like  programming  or  teaching 
or  instructional  design.  Still  others  were  instructors  who 
were  released  from  all  or  part  of  their  teaching  duties  in 
order  to  develop  PLATO  lessons.  It  seems  that  both  teaching 
(or  curriculum  design)  experience  and  subject  matter  exper- 
tise were  necessary  but  not  sufficient  qualifications  for 
good  authoring.  Instructors  who  had  only  lectured  and  had 
not  interacted  with  students  on  an  individual  basis  were  not 
adequately  sensitive  to  the  student's  needs.  They  tended  to 
equate  teaching  with  presenting  and  incorporated  very  little 
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interaction  in  their  lessons.  Skilled  programmers  who  had 
no  teaching  experience  produced  lessons  which  were  ineffec- 
tive. Retraining  carry-over  staff  was  relatively  unsuc- 
cessful. A director  who  decided  to  retrain  under qua  1 i f i ed 
carry-over  staff  later  expressed  some  misgivings.  Such 
staff  required  greater  than  average  consulting  assistance 
and  many  of  the  lessons  they  produced  were  poor  or  unuse- 
able.  In  another  incident  authors  who  had  previous  CBE 
experience  were  ret'"ained  to  use  the  TUTOR  language.  Many 
of  these  authors  were  unwilling  to  learn  about  the  new  and 
more  sophisticated  aspects  of  PLATO  or  to  apply  that  know- 
ledge to  producing  PLATO  lessons.  Instead  they  wrote  mate- 
rials patterned  after  the  inefficient  lesson  formats  which 
they  had  been  required  to  use  previously. 

When  instructors  were  given  released  time  to  write 
lessons,  they  needed  at  least  half-time  release  to  be 
effective.  Persons  with  very  limited  amounts  of  released 
time  spent  most  of  their  time  in  "warm-up"  or  overhead 
activities  and  contributed  little  to  a project.  Even 
half-time  release  was  not  always  effective.  The  intent  to 
provide  half-time  for  CBE  was  honest,  but  it  could  not 
always  be  honored.  Teaching  duties  differ  from  CBE 
responsibilites  in  the  immediacy  of  the  needs,  deadlines, 
and  payoffs.  When  the  total  work  load  became  excessive, 
teaching  duties  such  as  meeting  a class,  took  precedence 
over  lesson  writing.  Consequently  teaching  sometimes  took 
more  than  the  50%  time  allotted  and  CBE  lesson  writing 
suffered  accordingly.  On  the  other  hand,  full-time  release 
was  not  always  successful  because  the  instructors  became 
isolated  from  the  other  instructors  and  from  the  problems  of 
the  students. 

A decision  specific  to  the  military  environment  was 
whether  or  not  to  employ  civilian  staff.  The  decision  to 
choose  both  military  and  civilian  authors  worked  out  satis- 
factorily at  3 of  the  4 sites  observed.  At  the  fourth  site, 
pay  differentials  for  equivalent  work  caused  an  exodus  of 
military  authors. 

1 . Qualifications 

Case  2C1.1  - High  turn-over  subject  matter  experts 

The  director  at  one  site  was  not  free  to  choose  his  own 
personnel  to  author  CBE  lessons.  Most  of  his  staff  were 
subject  matter  experts  (SME)on  one-year  assignments  to  a 
training  organization.  The  SMEs  were  trained  in  TUTOR,  and 
programmers  were  hired  to  help  them  develop  lessons.  The 
programmers  were  effective  in  helping  them,  but  the  ratio  of 
1 programmer  to  4 authors  was  inadequate  to  meet  the 


authors'  needs.  The  lessons  oroduced  were  essential Iv  like 
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textbooks.  One  frame  after  another  was  an  extensive  display 
of  text.  Very  little  interaction  was  incorporated  in  the 
lessons. 

Case  2C1.2  - Teaching  experience 

In  another  project  personnel  had  already  developed  use- 
ful lessons.  A new  director  arrived  with  some  of  his  former 
staff  members.  They  had  teaching,  but  not  CPE,  experience. 
They  tried  to  make  CBE  fit  old  molds,  refusing  to  learn  from 
experience  of  existing  staff.  This  resulted  in  bad  rela- 
tionships between  the  old  and  new  staffs  and  ultimately  in 
inefficient  program  implementation. 

Case  2C1.3  - University  students  enrolled  in  CBE  course 

Students  enrolled  in  a university  course  for  the 
specific  purpose  of  learning  how  to  write  lessons  on  the 
PLATO  system.  Since  the  professor's  department  had  no  funds 
for  developing  PLATO  lessons,  he  decided  to  have  the 
students  write  lessons  that  could  be  used  for  a beginning 
course  in  his  subject. 

The  result  was  an  exceedingly  diverse  set  of  lessons  of 
varied  quality,  depending  on  the  abilities  and  motivations 
of  the  student  author.  Some  lessons  contained  gross  errors 
in  content.  Staff  members  were  able  to  get  lessons  revised 
for  content  accuracy,  but  it  was  often  difficult  if  not 
impossible  to  get  the  student  author  to  make  other  kinds  of 
revisions,  such  as  spelling,  or  more  reasonable  answer- 
judging.  The  first  time  the  lessons  were  used  for  a regular 
class,  the  CBE  students  indicated  that  they  enjoyed  using 
the  PLATO  system.  However,  they  did  not  do  as  well  as  the 
as  the  non-CBE  students  on  exams.  The  instructor  ascribed 
this  performance  difference  in  part  to  lesson  inadequacies 
and  errors. 

Epilog.  The  instructor  subsequently  decided  to  revise  the 
lessons  himself,  at  a considerable  investment  of  time. 
Classes  that  used  the  revised  versions  then  performed  as 
well  as  non-CBE  students. 

Case  2C1.4  - University  students,  extracurricular  activity 

Department  funds  were  not  available  for  developing  CPE 
lessons.  The  decision  was  made  to  have  student  programmers 
write  lessons.  The  students  seemed  to  learn  the  material 
they  programmed  themselves,  but  the  lessons  could  not  be 
used  to  teach  other  students.  An  instructor  finally  ended 
up  revising  some  of  the  more  promising  lessons  and  supple- 
menting them  with  lessons  written  on  his  own  time. 


?. . Special  considerations  in  military  environments 

Case  2C2.1  - Both  military  and  civilian  authors,  unsuc- 
cessful 

At  a military  base,  the  administrators  were  uncertain 
whether  civil  service  staff  or  military  staff  would  make 
better  authors.  They  decided  to  make  a small  research  study 
of  the  issue  and  to  select  half  the  staff  for  the  project 
from  each  and  to  study  the  differences  later.  The  civilians 
had  comparatively  precise  job  descriptions  and  a higher  pay 
scale  than  their  military  colleagues.  When  the  system  was 
"down",  traditional  course  authors  complained  to  the 
director  that  the  CBE  authors  were  just  "sitting  around."  So 
the  director  imposed  demeaning  clerical  and  janitorial  work 
on  the  CBE  staff  group  during  down  time.  The  civilians  were 
able  to  point  to  their  job  descriptions  and  thus  escape-- 
doubling  the  "nasty"  work  for  the  military  personnel.  This, 
combined  with  their  lower  pay,  created  a substantial  morale 
problem  for  the  military  authors.  Many  left  or  tried  to 
leave  before  the  project's  end--thereby  causing  substantial 
problems  and  delays  while  new  staff  were  identified  and 
trained . 

Case  2C2.2  - Both  military  and  civilian  authors,  successful 

At  two  other  military  sites,  civil  service  and  military 
personnel  worked  together  without  major  problems.  There 
were  apparently  small  differences  in  pay  and  virtually  no 
differences  in  duties  or  responsibilities. 

Case  2C2.3  - All  military  authors,  personnel  changes 

One  military  project  was  acquainted  with  the  situation 
described  in  Case  2C2.1.  So  they  decided  to  have  an  all 
military  staff.  Most  of  the  selected  staff  were  officers, 
and  thus  they  did  not  encounter  the  problem  of  demeaning 
janitorial  duties  being  foisted  on  them,  as  at  the  other 
site.  Obviously  no  mil itary-vs  . -civil ian  problems  arose. 
However,  this  project  had  to  deal  with  the  severe  problems 
which  any  group  which  includes  military  personnel  must  be 
prepared  to  accept.  That  is,  promotions  and  reassignment  of 
personnel  forced  reorganizations  of  the  CBE  project,  with 
the  attendant  loss  of  efficiency. 

Case  2C2.M  - All  military  authors,  problems  due  to  rank 

At  a military  site,  the  external  supportive  institution 
encouranged  the  director  to  include  instructional  designers 
and  evaluators  as  part  of  the  lesson  development  group. 
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Consequently,  an  experienced  lesson  designer  was  assigned  to 
assist  the  C^E  authors.  He  held  a lower  military  rank  than 
the  authors.  Even  though  he  was  very  competent,  and  the 
authors  would  have  written  better  lessons  if  they  had 
followed  his  advice,  they  largely  ignored  him  because  of  his 
lower  rank.  A different  instance  of  the  effect  of  military 
rank  is  given  in  Case  3A3.1  below. 

3.  Carry-over  staff  from  antecedent  projects 

Case  2C3.1  - Retraining  existing  staff  to  be  authors 

One  project  was  set  up  during  a period  of  tight  organi- 
zational funding  and  a shortage  of  local  manpower.  There 
were  no  funds  to  hire  new  staff  and  the  existing  pool  of 
unassigned  personnel  from,  which  to  staff  the  project  did  not 
contain  qualified  individuals.  The  project  director  was 
advised  he  had  two  choices:  employ  the  under-qualified 
staff  and  use  them  as  best  he  could,  or  employ  no  one  for 
those  positions  for  the  term  of  the  project.  That  is,  the 
positions  could  not  be  held  open  so  that  better  qualified 
staff  might  later  be  added. 

The  director  decided  tht  slow  producers  would  be  pref- 
erable to  understaffing  the  project  and  hence  filled  all 
his  positions. 

The  shortcomings  of  the  staff  forced  the  director  to 
revise  his  management  of  the  project.  He  devised  a team- 
oriented  approach  so  that  the  weaker  authors  could  serve 
mainly  as  subject  matter  experts.  One  person  was  removed 
entirely  from  programming/subject-matter  duties  and  given 
clerical/editorial  responsibilities  until  his  position  could 
be  terminated. 

Epilog . At  a later  time,  the  director  and  his  assistant 
expressed  some  misgivings  about  the  decision.  The  under- 
qualified staff  consumed  an  excessive  amount  of  the  time  of 
on-site  and  off-site  consultants.  Their  work  was  sometimes 
unacceptable  and  required  complete  reworking  by  more- 
qualified  staff.  But  the  fact  that  some  of  the  staff  were 
well-known  in  the  institution  probably  enhanced  the  accep- 
tance of  the  materials. 

Case  2C3.2  - Failure  to  adapt  to  new  features 

Some  members  of  the  staff  at  one  site  had  used  another 
CBE  system.  They  often  tried  to  force  their  PLATO  lessons 
into  old  molds  and  were  reluctant  to  use  more  sophisticated 
and  powerful  techniques  on  a new  system.  For  example,  they 
did  not  use  the  data-keeping  features  of  the  PLATO  system 
but  rather  devised  cumbersome,  less  informative,  but 
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familiar  routines  similar  to  those  which  had  been  used  on 
their  previous  system.  Some  staff  had  been  editors  of 
programmed  text.  Initially  they  attempted  to  use  CBE  as  a 
programmed  text  ignoring  the  possibility  of  using  feedback. 

'i  The  lessons  were  boring  and  ineffective. 

4.  Staff  from  within,  released  full-time 

; Case  2C^.^  - Isolation  from  other  instructors 

j Instructors  at  a college  were  given  full-time  for 

I production  of  materials  and  complete  freedom  in  setting 

objectives  although  they  had  no  prior  experience  in  either 
i instructional  design  or  CBF.  Although  they  became  profi- 

i cient  with  the  CBE  language,  their  total  lesson  productivity 

■ was  not  impressive.  Moreover,  they  were  perceived  by  other 

, members  of  their  departments  as  "outsiders"  and  had  diffi- 

culty in  scheduling  trial  use  of  their  materials  by  stu- 
dents. In  one  instance,  a person  produced  no  material  at 
all  during  a year  of  released  time  and  left  the  school  for 
another  position  near  tne  end  of  the  period. 

Case  2C4.2  - Isolation  from  students 

Kegt  .ar  instructors  were  released  to  be  CBE  authors  but 
they  had  almost  no  experience  in  any  aspect  of  CBE.  They 
tended  to  produce  material  without  attempting  to  test  it 
with  students  or  relate  it  to  student  needs.  The  rate  of 
production  was  also  rather  low  because  deadlines  were  in  the 
distant  future. 

Case  2Ci4.3  - Military  setting,  isolation 

A number  of  instructors  at  a site  were  chosen  to  become 
CBE  authors.  They  tended  to  be  the  younger  and  better  edu- 
cated of  the  group.  Those  who  were  not  picked  thought  that 
the  CBE  people  were  somewhat  of  an  "elite"  group  and  had  the 
easier  and  more  interesting  job.  The  CBE  staff  no  longer 
did  any  teaching  and  their  lesson  writing  became  isolated 
from  the  course.  The  lessons  they  produced  were  of  variable 
quality.  This  situation,  coupled  with  administrative  pro- 
blems, the  small  flow  of  students,  and  the  fact  that  there 
were  no  real  problems  teaching  the  course  in  the  traditional 
way,  resulted  in  little  use  of  the  lessons. 

At  another  site  with  a nearly  identical  situation,  one 
of  the  authors,  making  an  exit  interview,  stated  he  felt 
that  one  author  working  full-time  would  produce  less  than 
two  authors  working  half-time,  and  that  the  latter  would 
better  be  able  to  relate  the  lessons  to  the  student  audience 
and  to  smoothly  implement  the  lessons. 


22 


5.  Staff  from  within,  released  part-time 
Case  2C5.1  - No  CBR  experience 

in  2 college  level  courses,  instructors  with  no  prior 
instructional  design  or  CHE  experience  produced  little  mate- 
rial during  an  initial  year  of  507-released  time.  Put  they 
were  able  to  test  those  materials  within  their  own  courses 
and  develop  effective  instructional  approaches  which  led  to 
production  of  substantial  amounts  of  material  in  later 
years . 

Case  2C5.2  - CRH  experience 

in  three  college  level  courses,  instructors  who  had 
spent  two  or  more  years  developing  CHE  materials  on  their 
own  time  were  given  half-time  release  to  consolidate  and 
supplement  materials  already  produced.  Productivity  was 
quite  high  in  two  cases  and  in  all  cases  the  released  time 
resulted  in  substantial  additions  to  the  amount  of  material 
available  to  the  departments  involved. 

Case  2C5.3  - Less  than  half-time  release 

Regular  instructors  were  released  1/4  time  to  develop 
instructional  materials  for  CBE.  That  was  not  enough  time 
for  them  to  write  lessons.  The  best  they  could  do  was 
review  lessons  that  others  had  written  and  sometimes  they 
could  do  little  of  that.  Almost  nothing  useful  was  accom- 
plished . 

In  one  instance  a person  with  1/6  released  time 
reported  that  he  felt  that  he  had  made  no  contribution  at 
all  to  the  project.  This  view  was  shared  by  other  members 
of  the  project  who  felt  that  the  efforts  in  attempting  to 
train  him  had  wasted  limited  resources.  Attempts  to  avoid 
this  problem  resulted  in  a range  of  released-times  tried  by 
a number  of  projects. 

D.  Planning  the  Project 

In  order  to  schedule  CBE  lesson  production,  project 
directors  needed  information  on  which  to  base  plans.  The 
only  data  that  were  initially  available  were  based  on  a 
special  group  of  highly-experienced  workers.  "^heir  data 
were  cited  in  response  to  questions  from  directors  of  new 
projects,  but  the  fact  that  the  data  were  for  experienced 
workers  was  often  not  communicated  effectively.  Even  after 
more  complete  data  became  available,  figures  of  30  hours  of 
production  time  for  one  hour  of  student  contact  time  (valid 
for  one  set  of  very  proficient  authors)  were  cited  rather 
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than  fi(’,ureis  of  sovoral  hundt’ed  liour.s  per'  cont.aet  hour- 
(valid  for  new  authors  durinp,  their  first  year).  Conse- 
quently, planninr,  for  production,  when  done  at  all,  was 
often  predicated  on  production  rates  that  were  impossible 
and  unrealistic  for  the  untrained  workers  available  to  new 
projects , 

Some  directors  made  overatnbitious  plans,  which  led  to 
failure  to  meet  deadlines.  In  cases  where  no  continRencies 
had  been  planned  and  deadlines  could  not  be  met,  the  deci- 
sions of  project  directors  varied  from  just  doinR  the  best 
they  could  under  the  circumstances  to  giving  up  lesson 
writing  entirely  and  writing  routers  for  managing  lessons 
which  had  been  written  by  others. 

Administrative  organizations  of  the  projects  also 
varied  considerably.  Some  projects,  particularly  in  the 
military,  brought  together  a number  of  individuals  to  pro- 
duce a fixed  number  of  lessons  in  a given  length  of  time. 
Directors  tended  to  underestimate  the  time  that  would  be 
needed  and  some  also  failed  to  arrange  for  contingencies  in 
case  they  would  be  unable  to  meet  their  goals.  The  suc- 
cessful projects  were  the  ones  which  were  planned  so  that 
lessons  could  be  effectively  used  even  if  goals  were  not 
completely  met. 

Some  projects  evolved  gradually,  with  a single  indivi- 
dual first  Raining  experience  writing  individual  lessons, 
and  later  on  assembling  more  staff  members  and  creating  a 
curriculum.  Many  of  these  were  highly  successful  projects. 

Another  administrative  question  that  had  to  be 
addressed  by  a CBE  project  director  was  liow  much  planning  to 
do  ahead,  and  how  much  to  leave  open  and  subject  to  the 
discretion  of  the  author.  This  was  particularly  relevant 
for  projects  that  used  the  PLATO  system,  where  it  was  a 
simple  matter  to  "compose"  and/or  revise  lessons  at  the 
terminal.  The  system  does  not  constrain  the  lesson 
developer  to  make  long  range  commitments  to  a fixed  plan 
which  will  be  difficult  to  change  at  a later  time.  Early 
projects  varied  from  detailed  planning  to  some  general 
planning  with  just  a few  details  prespecified,  to  almost 
complete  improvisation.  The  two  extremes  were  apparently 
the  least  productive  in  the  long  run. 

The  selection  of  appropriate  media  was  an  important 
consideration  in  project  development.  CFIF  is  not  always  the 
most  suitable  medium  for  all  instruction.  When  project 
directors  used  PI.ATO  instruction  exclusively  without  consid- 
ering the  appropriateness  of  such  use,  the  CBE  lessons  were 
frequently  ineffective,  boring,  and  not  cost  effective. 
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1.  Production  p.oals 

Case  PD1.1  - Working  from  the  end  of  course  to  beginning 


At  one  military  site,  plans  were  drawn  up  even  before  i 

training  was  begun  in  the  programming  language.  “lince  a j 

standard  course  of  instruction  was  already  in  use,  the  task  i 

of  the  project  was  to  develop  parallel  CBF  instruction.  The 
decision  was  made  to  write  the  last  lesson  of  the  course 
first  and  then  continue  to  write  in  the  reverse  order  in 
which  the  lessons  would  be  used.  Several  advantages 
resulted.  The  student  did  not  have  to  wait  for  a lesson 
until  the  whole  course  was  ready.  He  simply  worked  in  the 
standard  mode  until  he  reached  the  point  where  the  CBH 
lessons  were  ready,  and  then  continued  with  them  until  the 
end.  There  were  other  advantages  to  this  plan.  The  first 
CBE  lesson  the  student  encounters  sets  his  attitude  toward 
the  medium.  Since  the  author's  first  lessons  are  apt  to  be 
his  worst,  it  is  best  if  if  they  are  not  the  first  CRE 
lessons  the  student  encounters.  In  the  plan  under  discus- 
sion, the  student  did  not  encounter  the  author's  first  les- 
sons until  the  end,  by  which  time  he  was  hopefully  tolerant 
of  its  shortcomings  and  more  adapted  to  the  medium. 

Finally,  evaluation  was  possible  even  though  not  all  of  the 
lessons  originally  planned  were  completed. 

Case  2D1.2  - Fixed  number  of  hours,  no  contingencies  ; 


Some  projects  were  given  some  latitude  in  setting  their 
own  goals.  Thus,  they  were  free  to  set  goals  in  terms  of 
topics  to  be  covered  or  in  terms  of  student  contact  hours 
they  would  provide. 

In  two  instances,  inexperienced  project  directors  set 
goals  far  beyond  the  resources  of  their  groups  and  failed  to 
make  any  attempt  to  provide  alternative  goals  should  the 
planned  ones  prove  impossible.  Despite  continued  slippage 
of  deadlines,  the  planned  goals  were  maintained  until  final 
deadlines  were  so  close  that  it  was  obvious  to  all  that 
there  was  a real  chance  for  total  failure  of  the  projects. 

As  a last-minute  alternative,  all  attempts  at  lesson  prod- 
uction were  abandoned  and  lessons  written  by  other  groups 
were  adopted  (despite  the  fact  that  many  of  these  materials 
had  been  regarded  with  contempt  a few  months  earlier). 
Ultimately,  the  adopted  materials  were  arranged  in  well- 
documented  packages  that  were  accessed  by  specially  designed 
routers.  While  the  final  "product"  was  a reasonable  outcome 
of  the  project,  it  would  probably  have  been  of  far  higher 
quality  had  the  original  goal  been  altered  much  earlier. 


L. 
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Case  2D1.3  - Inappropriate  basis  for  production  schedule 

The  decision  was  made  to  develop  lessons  which  would 
supply  a specified  number  of  student  contact  hours.  Plans 
were  erroneously  based  on  planning  guidelines  which  applied 
to  experienced  rather  than  the  inexperienced  authors 
employed  on  this  project.  As  a result,  the  director  grossly 
underestim.ated  production  time  requirements.  He  further 
failed  to  provide  alternative  plans  should  the  original 
goals  prove  impossible  for  any  reason.  When  it  became 
obvious  that  the  original  goals  were  unattainable,  they  were 
substantially  scaled  down  (and  contingency  plans  were  made 
for  future  problems).  This  alteration  of  plans  under  pres- 
sure of  potential  failure  (rather  than  as  part  of  a planned 
sequence  of  alternative  goals)  was  demoralizing  to  all  par- 
ticipants of  the  project  and  tended  to  lead  project  direc- 
tors to  engage  in  "cover-up"  and  blame-shifting  tactics 
rather  than  in  activities  directed  toward  improving  instruc- 
tional quality. 

Case  2D1.4  - A complete  set  of  lessons 

Flans  for  one  project  entailed  a new  curriculum  to  be 
developed  on  a new  medium,  CBE.  All  the  lessons  in  the 
course  had  to  be  completed  in  order  to  judge  its  effec- 
tiveness. This  virtually  eliminated  the  possibility  of  a 
"partial"  success  since  unless  all  the  lessons  were  com- 
pleted there  could  be  no  project  evaluation.  When  it  became 
apparent  that  all  of  the  lessons  would  not  be  completed  by 
the  deadline,  the  entire  project  was  scrapped,  officially 
for  other  reasons.  However,  the  impending  failure  to 
complete  the  lessons  was  a major  contributing  factor. 

Case  2D1.5  - Incorporate  lessons  as  they  become  available 

In  at  least  three  projects,  the  production  goals  were 
specified  in  terms  of  lessons  covering  particular  subject 
matter  topics.  At  first  the  project  director  was  the  only 
author.  As  soon  as  lessons  were  "finished"  they  were  incor- 
porated into  courses  in  appropriate  places.  Eventually, 
the  director  added  some  staff  members.  Although  completion 
deadlines  were  not  strictly  pre-specified , production 
progressed  well.  This  was  possible  because  the  directors 
themselves  were  competent,  highly  motivated  authors  who 
stayed  in  constant  contact  with  their  staff.  They  set 
realistic  goals  but  in  an  informal,  flexible  setting. 
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12.  Prespecification  of  all  routing 

;;  Case  2D2.1  - Isolation  of  instructor  from  C'lF 

^ ’ Curriculum  materials  were  written  at  a development  cen- 

[.  ter  for  implementation  elsewhere.  The  project  director  had 

I the  alternatives  of  making  the  procram  completely  self- 

l 1 contained  or  requiring  instructor  intervention.  He  chose 

I the  former.  He  therefore  decided  that  the  entire  curriculum 

■ .1  had  to  be  mapped  out  before  any  student  trials  were 

I ' attempted.  This  included  writing  every  objective  and 

routing  contingency.  Since  the  full  weight  of  instruction 
j ; fell  on  the  CBE  materials,  a great  deal  of  time  had  to  be 

I ' spent  by  the  staff  in  altering  and  maintaining  the  elaborate 

! ' routing  and  branching  structures.  They  did  not  have  time  to 

; • revise  the  lessons  themselves  or  to  add  new  ones  where 

• necessary.  Since  the  lessons  had  minimal  pretesting  with 

students,  they  often  needed  a good  deal  of  revision  if  they 
were  to  be  effective.  The  instructors  felt  that  since  the 
router  was  doing  all  the  work,  they  did  not  need  to  get 
involved.  When  problems  arose  they  did  not  try  to  find 
■ solutions  themselves,  but  rather  called  on  the  CBE  staff. 

Furthermore,  they  felt  frustrated  because  the  lessons  were 
not  meeting  the  needs  of  the  students,  and  there  was  nothing 
they  could  do  about  it. 

l' 

J bpi log . The  router  was  scrapped  and  a new  one  was  written. 

I This  one  made  it  the  teacher's  duty  to  select  lessons  for 

'i  the  students.  The  task  for  the  teacher  was  made  as  simple 

as  possible.  Although  this  made  some  extra  work  for  the 
teachers,  they  became  increasingly  involved  in  CPE  as  a reg- 
ular part  of  instruction.  Also,  as  they  became  more  familiar 
with  the  terminals  and  felt  more  at  ease  with  them,  they 
began  to  try  to  solve  some  of  the  problems  themselves.  The 
instructors'  calls  for  help  from  the  CBE  staff  decreased 
noticeably,  and  it  became  possible  for  the  CBE  staff  to 
concentrate  more  on  improving  the  instructional  quality  of 
the  materials. 

3.  Failure  to  define  goals 

s Case  2D3.1  - No  clear  goals 

At  one  site,  authority  for  decisions  on  project  goals 
( was  divided  between  two  groups.  It  took  four  months  to  get 

' approval  of  an  initial  set  of  plans.  During  that  time  the 

; student  population  changed.  Moreover,  the  plans  turned  out 

to  be  impractical  due  to  lack  of  experience  with  CBE  at  the 
time  they  were  written.  New  plans  had  to  be  formulated.  A 
major  obstacle  to  approval  was  a conflict  in  the  goals  of 
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the  two  groups.  One  was  interested  in  seeing  CDF!  deliver 
instruction;  the  other  was  research  oriented.  The  project 
thus  went  along  without  clear  objectives.  Lack  of  decisive- 
ness resulted  in  frequent  changes  of  plans.  For  all  practi- 
cal purposes,  every  time  a plan  was  changed,  the  progress  in 
the  previous  period  was  cancelled.  The  net  effect  was  very 
limited  progress  for  the  project. 

4 . Media  selection 

Case  2D4.1  - Failure  to  be  selective 

At  several  sites  the  project  directors  assumed  that 
since  CRE  was  to  be  used,  it  had  to  be  done  to  the  exclusion 
of  other  media  or  had  to  be  incorporated  somehow  whether 
feasible  or  not.  At  one  site,  the  most  effective  medium  for 
a task  was  videotape,  so  the  project  director  decided  to  use 
PLATO  by  having  it  control  the  videotape.  In  another 
instance,  an  author  suggested  that  a particular  lesson  be 
presented  as  a handout,  but  the  project  director  decreed 
all  lessons  must  be  in  CBE.  The  result  was  a lesson  that 
was  boring  and  costly. 


Chapter  3 

PKUJhCT  DrVELCPMEf.'T 


A.  Director's  Role 

As  PLATO  projects  evolved,  sorr.e  continued  under  the 
leadership  of  the  initiator.  Others  underwent  one  or  more 
changes  of  directors.  In  many  larger  projects,  the  organi- 
zation necessitated  several  levels  of  authority  and  in  those 
cases  administrative  structure  varied.  For  example,  within 
a single  institution,  CBE  lessons  were  being  written  in 
several  subject  areas.  Each  group  had  its  own  project 
leader  and  all  such  projects  were  under  the  supervision  of  a 
higher  level  administrator.  In  some  cases  the  top  level 
director  was  also  the  director  of  one  of  the  sub-projects; 
in  others  he  was  essentially  a coordinator  and  administrator 
in  management  rather  than  in  curriculum  development. 

Unfortunately,  roles  and  domains  of  authority  were  not 
always  prespecified  in  these  multi-level  authority  groups. 
Leaders  at  the  same  or  parallel  levels  of  authority  could 
and  sometimes  did  intrude  on  the  projects  of  ethers,  giving 
conflicting  directives.  When  this  happened,  staff  were 
confused  and  progress  was  hampered. 

As  noted  above,  leaders  who  chose  to  be  in  constant 
contact  with  projects  were  a positive  force  in  successful 
development  and  could  often  prevent  problems  from 
developing . 

Project  progress  was  sometimes  slowed  due  to  a change 
of  directors.  In  at  least  two  cases,  this  happened  because 
the  new  director  ignored  the  accumulated  knowledge  of  his 
predecessor  and  started  from  scratch. 

A special  problem  in  the  military  situation  arose  when 
promotions  raised  a staff  member  to  a rank  above  that  of  the 
project  director.  The  new  senior  officer  assumed  responsi- 
bility according  to  military  code,  and  the  structure,  roles, 
and  responsibilities  previously  set  up  were  left  in 
disarray. 

1 . Keeping  in  touch 

Case  3A1.1  - Positive  result  of  support 

At  one  large  institution,  CBE  lessons  were  being  devel- 
oped in  several  departments.  The  top  administrator,  who  had 
overall  responsibility  for  the  individual  projects,  chose 
staff  for  each  project  and  kept  in  personal  contact  with  the 
staff  and  projects.  This  was  a major  factor  in  helping  him 
develop  a viable  CBE  group  in  at  least  one  department. 
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Case  3A'I*2  - Intrusion 

A number  of  projects  were  headquartered  at  a curriculum 
development  center.  Authors  worked  in  proximity  to  those 
from  other  groups  as  well  as  their  own.  A high-ranking 
member  of  the  center  staff  took  a well-intentioned  interest 
in  projects  for  which  he  had  no  responsibility  and  made 
suggestions  which  were  interpreted  as  orders  and  which 
sometimes  conflicted  with  those  of  the  real  leader.  The 
result  was  confusion  on  the  part  of  project  members,  divi- 
sive power  struggles  in  which  staf?'  members  pitted  the  word 
of  one  leader  against  that  of  the  other,  and  lower  produc- 
tivity . 

2.  Effect  of  changing  directors 
Case  3A2.1  - Reinventing  the  wheel 

in  two  cases,  after  the  project  had  been  under  way  for 
a year  or  more,  the  director  left  and  a new  one  was 
appointed.  When  the  new  director  arrived,  he  began  the 
project  anew,  as  if  nothing  had  been  done  previously.  He 
was  not  interested  in  learning  from  or  building  on  previous 
experience.  Often  these  new  directors  brought  in  their 
own  staff  at  better  pay  than  the  former  staff  even  though 
the  new  staff  was  inexperienced  in  the  skills  needed  for  CBE 
design  and  required  training  by  the  old  staff.  Morale  and 
productivity  of  the  old  staff  fell,  many  mistakes  were 
repeated,  and  time  was  wasted. 

3.  Military  role  conflict 

Case  3A3-1  - Military  role  conflict 

The  director  of  a CBE  project  at  a military  site  was 
chosen  for  his  background  and  experience  in  administration. 
He  was  hired  early  in  the  project  before  other  staff  were 
present.  He  designed  a structure,  established  roles,  and 
began  the  project.  Several  months  later,  one  of  the  staff 
members,  an  author,  was  promoted  (on  the  basis  of  previous 
service)  to  a higher  rank  than  that  of  the  project  director. 
As  the  ranking  officer,  military  code  held  him  responsible 
for  the  operation.  Role  distinctions  became  confused  as  the 
lines  of  authority  were  rearranged.  Personality  differences 
between  the  two  leaders  added  to  the  stress.  The  original 
leader  was  greatly  liked  and  respected  by  the  staff  while 
the  new,  ranking  leader  was  disliked.  The  continuing  pre- 
sence of  the  former  director  on  the  project  also  added  to 
the  confusion  of  some  staff  members. 
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Case  3A3-2  - Non-supporti ve  aid 

'*  One  consultant  who  was  a pood  programmer  and  who  had 

t done  most  of  his  consulting  with  persons  interested  in  com- 

' •'uters  rather  than  instructional  design  was  highly  critical 

of  anyone  who  did  not  readily  grasp  the  CBE  language.  One 
trainee  who  ultimately  turned  out  to  be  a very  effective 
instructional  designer  was  ready  to  quit  after  being  told 
that  he  would  "never  be  a good  author"  by  this  consultant. 

A more  supportive  consultant  who  was  more  oriented  toward 
I instructional  design  was  able  to  salvage  the  situation  at 

the  last  minute  . 

B.  Staff  Considerations 

I 

The  software  staff  for  PLATO  IV  declined  to  make 
available  a printed  manual  about  the  TUTOR  language.  Such  a 
document  had  been  prepared  for  PLATO  III.  However,  since 
the  system  was  continually  evolving  and  changing,  the  manual 
soon  became  incomplete  and  parts  of  it  became  obsolete.  A 
hard  copy  manual  for  PLATO  IV  would  have  had  the  same 
shortcomings.  It  was  precisely  these  two  factors,  the 
evolutionary  nature  of  the  programming  language  and  the  lack 
of  a printed  manual,  that  made  learning  TUTOR  particularly 
problematical  for  new  authors.  Consequently  a strong  sup- 
portive environment  was  important  and  contributed  heavily  to 
the  progress  of  successful  authors.  Many  of  the  best 
authors  were  trained  in  an  apprenticeship-type  relationship 
with  experienced  authors.  Other  projects  found  that  author 
productivity  was  improved  when  the  services  of  an  on-site 
consultant  were  available.  In  several  incidents,  where 
would-be  authors  made  slow  or  little  progress,  part  of  the 
difficulty  could  be  traced  to  the  lack  of  psychological 
support.  They  were  expected  to  learn  authoring  by  them- 
selves without  the  regular  guidance  or  consulting  services 
of  an  experienced  individual. 

Some  projects  were  organized  to  include  lesson 
designer-programmer  teams.  When  the  programmers  were 
students,  hired  on  an  hourly  basis,  personnel  turnover  was 
high  and  progress  was  impeded.  A more  successful  arrange- 
ment resulted  when  the  programmer  was  treated  like  a profes- 
sional: held  a regular  appointment,  (preferably  at  least 

2/3  time)  and  regarded  the  work  as  a job  with  top  priority 
over  other  activities.  One  other  factor  that  fostered 
smooth  relationships  was  the  decision  that  the  lesson 
designer  think  the  lesson  through  carefully  and  perhaps 
design  a few  displays  before  turning  it  over  to  the  pro- 
grammer. This  system  minimized  the  number  of  revisions  that 
were  needed  later.  The  programmer  no  longer  felt  that  all 
of  his  previous  work  was  wasted  because  total  rewrites  were 
rarely  necessary. 
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1.  The  need  for  a supportive  environment 
Case  3B1 . 1 - Apprenticeship  relationship 

in  two  subject  areas,  project  directors  initially 
developed  their  own  CBE  lessons.  After  these  lessons  were 
in  use  by  students,  the  directors  hired  additional  personnel 
to  write  lessons.  They  did  not  provide  a formal  training 
course.  Instead,  the  new  authors  reviewed  many  of  the 
existing  lessons  to  get  some  concept  of  the  scope  of  the 
lessons  and  the  capability  of  the  CBE  system.  Then  they 
learned  the  TUTOR  language  on  their  own.  When  they  began 
writing  their  lessons,  they  worked  in  an  apprenticeship 
relationship  with  the  director  and  other  experienced 
authors.  The  new  staff  member  could  rely  on  his  m.entor  as 
so  =!one  who  really  understood  the  complexities  of  CBE 
authoring  and  who  was  supportive  of  his/her  tr ial-and-error 
efforts  as  well  as  initial  successes  and  failures.  This 
kind  of  relationship  was  a positive  factor  for  the  author's 
morale  and  productivity. 

Case  3B1.2  - Senior  programmer  as  part-time  consultant 

In  several  groups  a senior  programmer  was  available  for 
quick  personal  consulting  for  staff  who  were  learning  TUTOR. 
This  enabled  them  to  make  reasonable  progress  while 
learning.  It  also  prevented  needless  frustration.  In  addi- 
tion, the  experienced  person  could  guide  the  new  author  by 
Indicating  which  parts  of  the  language  to  learn  immediately 
and  which  to  leave  for  a future  time. 

Case  3B1.3  - Different  roles  of  consultants 

One  project  manager  decided  that  his  group  needed  the 
services  of  an  on-site  programming  consultant,  so  he  hired 
one.  The  authors  were  glad  to  have  this  help  and  lesson 
development  progressed  well.  However,  the  manager  expected 
the  consultant  to  also  serve  as  a monitor  to  improve  lesson 
quality  . The  manager  was  disappointed  that  the  consultant 
did  not  assume  this  additional  responsibility. 

At  another  time,  a member  of  the  CERL  staff  provided 
part-time  consulting  services.  He  worked  at  a personal, 
rather  than  at  a group  level.  This  was  necessary  because 
the  frequent  change  in  leadership  meant  that  the  only  long- 
term (months)  relationships  that  could  be  formed  were  with 
the  authors.  Eventually  the  consultant's  opinions  and 
advice  were  used  as  the  basis  for  the  struggles  for  leader- 
ship. When  a final  management  structure  was  established, 
the  consultant  was  expelled  because  of  the  perception  that 
his  role  and  influence  were  disruptive. 
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Case  301-^  - -Jo  proprammer  consultant 

A site  director  obtained  four  PLATO  terminals  because 
he  wanted  to  provide  access  for  Lot.h  staff  members  and 
students.  He  verbally  encourap.eil  use  of  the  medium,  but 
decided  to  save  money  by  not  hiring  anybody  with  the  speci- 
fic responsibility  of  providinp  propramminp  service  and 
assistance  in  learning  about  the  system.  The  result  was  a 
very  limited  development  of  expertise  at  the  site.  1)  The 
staff  who  were  using  the  Pl.ATO  system  had  little  free  time 
and  felt  no  responsibility  or  inclination  to  program  for 
others.  2)  The  undergraduate  programmers  had  little  feel 
for  teaching,  little  time  available  for  hire,  and  high  turn- 
over. As  a consequence,  at  least  one  mini-course  slated  for 
implementation  on  the  PLATO  system  was  put  on  other  media 
because  no  dependable,  rapid  programming  aid  was  available. 

The  CDE  center  was  used  rather  casually  for  learning  how  to 

use  another  language  and  for  recreation.  It  was  used  for  : 

actual  CRE  training  by  those  few  instructors  who  had  written 

lessons  or  who  knew  how  to  access  materials  written  by 

others. 

Case  3R1*5  - Insufficient  references  and  no  training  program 

Shortly  after  the  implementation  of  the  PLATO  IV 
system,  several  educational  institutions  sent  full-time 
instructors  to  CFRL  to  learn  the  TUTOR  language  and  to  write 
lessons.  No  formal  training  program  was  available,  nor  was 
any  printed  material  available  for  reference.  At  that 
time  there  was  also  a shortage  of  terminals,  so  that 
trainees  could  not  always  get  one  when  they  needed  it.  They 
had  to  share  facilities  with  experienced  authors  who  often 
gave  them  personal  help  but  also  sometimes  left  the  trainees 
with  the  feeling  that  they  were  intruding.  Because  the 
staff  at  CERL  did  not  yet  understand  the  need  to  provide 
training  and  consulting  support  for  outsiders  who  were 
novices,  they  largely  ignored  this  group  of  would-be 
authors.  The  person  who  was  assigned  to  give  this  support 
had  other  full-time  duties  and  could  not  devote  the  time 
needed  to  teach  them.  Some  documentation  of  the  language 
was  available  on  line,  but  it  was  intended  to  be  used  as  a 
reference  for  experienced  authors.  It  did  not  fulfill  the 
needs  of  these  beginning  authors.  They  felt  growing  frus- 
tration at  seeing  things  that  they  wanted  to  program  hut 
were  unable  to.  The  result  was  that  at  the  end  of  the  year 
little  useable  material  had  been  produced  and  these  indivi- 
duals were  dismayed  at  how  long,  it  took  to  prepare  lessons. 
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2.  Lesson  desipner,  programmer  relationship 
Case  11)2.'  - Inadequate  pre-[)lanninp,  by  designer 

The  leaders  of  one  project  were  experienced  in  CDf  les- 
son design,  subject  matter,  and  teaching,  the  tarj’et  popula- 
tion. When  they  decided  to  use  the  team  approach  to  lesson 
development  and  hire  programmers,  they  made  inquiries  about 
potential  problems  in  this  arrangement.  One  of  the  most 
commonly  cited  was  the  fact  that  some  lesson  designers 
assigned  a task  to  a programmer  without  "enough"  careful 
consideration.  After  the  programmer  had  invested  a consi- 
derable amount  of  time  preparing  the  lesson,  the  designer 
would  frequently  decide  that  wasn't  what  he  wanted  after 
all,  and  would  scrap  the  lesson  for  another  plan.  This  was 
bad  for  morale;  programmers  felt  their  time  had  been  wasted. 
This  director  therefore  made  it  a policy  to  try  not  to 
assign  a lesson  for  programming  until  it  was  carefully 
thought  out.  Sometimes  the  lesson  designer  did  a few  dis- 
plays first  to  see  what  it  would  look  like  or  to  demonstrate 
what  he  wanted.  The  result  was  that  lessons  were  seldom 
discarded  and  the  programers  did  not  feel  they  were  wasting 
time.  On  the  rare  occasions  when  the  lesson  was  not  suc- 
cessful during  student  trials,  and  had  to  be  revised  consid- 
erably, the  programmers  understood  and  morale  did  not 
deteriorate  . 

Case  3h2.2  - Professional  part-time  programmers 

One  group  hired  student  programmers  on  an  hourly  basis. 
They  found  that  this  was  not  satisfactory  because  there  was 
a high  turnover  in  personnel.  This  meant  considerable  time 
had  to  be  spent  orienting  new  people  to  the  particular  plans 
and  conventions  of  their  project.  Moreover,  students  who 
worked  less  than  half  time  sometimes  found  they  did  not  have 
enough  time  to  fulfill  all  of  the  commitments  they  had  made. 
Studies  and  other  activities  took  priority  over  their  pro- 
gramming job.  The  decision  was  made  to  hire  "professional" 
programmers:  that  is,  people  who  considered  this  work  as  a 

job,  and  who  could  devote  at  least  half  time  to  it,  but 
preferably  2/3  time.  They  stayed  with  the  project  longer 
and  were  more  productive. 

3.  Difficulty  adapting  to  CRF 

Case  3P;3.1  - CHL  different  from  classroom 


tiew  authors  who  had  considerable  classroom  experience 
found  it  difficult  to  adju.st  to  CBF.  They  tried  to  model 
their  CRE  lessons  on  instruction  presented  in  the  classroom. 


They  failed  to  take  into  account  the  differences  between 
these  two  modes. 

In  the  classroom  these  instructors  used  the  discovery 
approach  to  learning,  guiding  the  students  by  asking  perti- 
nent questions.  Although  some  of  the  students  could  not 
answer  the  questions  themselves,  the  brighter  students  did. 
Thus  the  teacher  did  not  have  to  provide  the  answers  because 
classroom  interaction  enabled  the  slower  students  to  learn 
from  faster  ones.  In  the  CBE  lessons,  however,  all  of  the 
interaction  was  between  student  and  lesson.  Therefore,  more 
lesson  guidance  was  necessary  in  using  the  CBE  system  than 
in  the  classroom.  Considerable  time  was  wasted  writing 
unsuccessful  lessons  which  provided  inadequate  guidance  and 
feedback  for  the  students.  When  the  authors  observed 
student  trials,  they  found  the  students  openly  rebellious. 
The  authors  decided  to  revise  the  lessons.  They  provided 
more  explicit  instructions  and  adapted  the  difficulty  level 
to  the  needs  of  the  students. 

4.  Arrival  of  staff 

Case  3B4.1  - Staggered  arrival 

in  preparing  for  a large  project-oriented  development 
site,  the  majority  of  the  authoring  staff  were  given 
training  at  CERL  and  then  sent  on  to  a given  site  to  write 
lessons.  Six  months  later,  a second  group  of  staff  (who  had 
been  selected  earlier,  but  were  not  then  available)  arrived 
at  this  site.  The  author  training  for  the  new  authors  was 
slightly  re-oriented  to  adjust  for  what  were  perceived  by 
the  trainers  to  be  weaknesses  in  the  backgrounds  of  the 
first  group  of  authors.  In  particular,  instructional  design 
training  was  emphasized.  When  the  second  group  returned  to 
the  project  site,  they  considered  themselves  missionaries  tc 
those  who  were  already  established.  Conversely,  the  old- 
timers  had  already  formed  ideas  about  how  to  do  things  and 
considered  the  second  group  novices  whose  opinions  should  be 
taken  lightly.  Furthermore,  decisions  made  in  the  first 
part  of  the  project  were  not  not  always  explained  to  the 
second  group.  Friction  resulted  and  the  second  group  failed 
to  become  assimilated  into  the  first. 

5 . Job  security 

Case  3B5.1  - Low  morale 

In  at  least  three  cases,  the  staff  knew  that  the  pro- 
ject was  scheduled  to  run  out  of  funds.  The  directors  gave 
no  indication  of  whether  further  funds  might  be  forthcoming 
or  whether  other  projects  might  employ  them  when  the  current 
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project  ended.  Since  the  staff  did  not  know  whether  their 
jobs  would  still  be  available,  they  lost  motivation  and 
, wasted  a considerable  amount  of  time.  Often,  project 

reports  and  other  deadlines  could  be  achieved  only  if  the 
*;  entire  staff  worked  up  to  the  last  day  of  the  project.  in 

some  cases,  when  faced  with  the  prospect  of  impending 
failure,  many  staff  members  obtained  other  positions  well 
before  the  end  of  the  project  and  left  early.  As  a result, 
projects  were  completed  well  beyond  deadlines,  or  in  a 
slip-shod  fashion. 

[ t C.  Models  of  Organization 

Mahler  and  his  colleagues  (1976)  reported  that  each  of 
1 the  16  projects  they  interviewed  had  a unique  organization. 

Oenerally,  these  authors  found  four  broad  categories  for 
classification.  These  were:  independent  developer,  a 
col leagueship , a lesson  designer  with  programming  assis- 
[ tants,  and  some  combination  of  these.  Many  of  the  groups 

f modified  or  changed  their  organization  in  response  to  expe- 

I rience  and  to  the  shifting  needs  of  the  project.  Mo  one 

I plan  was  good  for  every  group.  Each  had  advantages  and 

I disadvantages. 

j Karly  in  the  development  of  the  PLATO  system,  authors 

were  mostly  professors  who  learned  TUTOR.  Some  of  them 
became  proficient  programmers  and  produced  lessons  readily. 
When  they  hired  new  staff  members  the  new  staff,  too,  became 
independent  authors.  All  liked  this  system  because  it 
i enabled  them  to  experiment  with  a wide  variety  of  teaching 

I techniques  and  ways  of  using  TUTOR.  It  did  not  force  them 

into  a predetermined  lesson  structure,  but  enabled  them  to 
observe  results  and  to  take  newly-gained  information  into 
account  for  succeeding  lessons.  Other  professors  found  that 
they  were  unable  to  produce  lessons  as  rapidly  as  they  would 
have  liked--for  example,  because  of  difficulties  in  pro- 
gramming efficiently--and  hence  hired  staff  whom  they  could 
direct  to  do  those  things  they  themselves  would  do  if  time 
and  efficiency  were  not  important. 

Following  the  export  of  the  PLATO  IV  system  from  the 
university  environment,  "instant"  versions  of  the  first 
approach  were  tried  with  varying  success.  For  example, 
several  staff  were  hired  and  set  to  work  using  the  inde- 
pendent approach  described  above.  Without  the  experience 
I built  up  from  several  years  of  exploratory  use  of  CP'^,  these 

efforts  were  not  overly  successful  in  creating  entire  curri- 
cula. The  authors'  independence  and  lack  of  ^^xperience 
resulted  in  wide  variations  in  lesson  styles  and  quality. 

The  shift  to  a team  approach  met  the  needs  of  some 
projects  hut  it  was  abandoned  or  modified  by  others.  Its 
success  depended  on  the  qualifications  of  the  staff  and 
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the  goals  of  the  projects.  In  three  different  projects, 
when  programmers  were  hired  to  code  for  lesson  designers, 
they  became  overly  creative.  They  embellished  and  imple- 
mented the  lesson  in  a way  that  was  neither  specified  nor 
intended  by  the  designer. 

in  cases  where  the  programmers  were  part-time  graduate 
students,  scheduling  problems  resulted  in  inefficient  pro- 
duction rates.  Some  directors  decided  to  return  to  indivi- 
dual authoring  and  to  depend  on  consultants  to  help  them 
with  particular  problems. 

In  two  projects,  a team  approach  became  necessary 
because  of  the  varied  quality  of  the  lessons  or  the  distri- 
bution of  talent  within  the  authoring  staff.  Both  teams 
produced  adequate  lessons  within  the  allotted  time.  Factors 
contributing  to  this  success  were  the  prespecification  of 
procedures,  experienced  management,  and  the  decision  to 
limit  the  instructional  strategies.  A disadvantage  of  the 
team  approach  in  one  incident  was  the  disproportionate 
amounts  of  work  by  the  members. 

Individual  authors  had  a personal  investment  in  their 
lessons.  They  were  often  defensive  about  them  and  reluctant 
to  accept  advice  or  make  revisions.  Use  of  the  team 
approach  reduced  this  problem  since  each  person's  "stake" 
and  involvement  in  the  lesson  was  not  so  great. 

1 . Lesson  designer  with  programmer  assistant 

Case  3C1.1  - Programmers  too  creative  (1) 

A project  head  was  a content  expert  and  university 
instructor.  He  learned  the  TUTOR  language,  but  felt  that 
his  time  could  be  more  profitably  spent  if  he  designed  the 
lessons  and  hired  a programmer  to  code  them.  However,  the 
programmers  were  carried  away  with  personal  creativity  and 
felt  compelled  to  contribute  to  the  lesson.  Each  of  the 
programmers  hired  created  lessons  which  were  not  as  intended 
or  specified  by  the  designer.  Sometimes  they  "embellished" 
the  lesson  with  graphics  and  animations  that  were  fun  to 
create. The  problem  was  that  they  did  not  improve  the  lesson. 
In  fact,  the  students  became  frustrated  when  they  had  to 
wait  the  few  seconds  until  the  end  of  the  graphics  before 
they  could  proceed  v^ith  the  lesson.  Sometimes  the  pro- 
grammer changed  the  presentation  slightly  but  the  total 
effect  was  different  from  that  intended  and  the  lesson  was 
apt  to  be  unuseable. 

Case  3C1.2  - Programmers  too  creative  (2) 

The  director  of  a large  curriculum  project  hired  staff 
members  who  were  experienced  teachers  and/or  lesson 
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designers.  But  many  of  them  felt  that  their  propramminp 
skills  were  insufficient  to  implement  their  ideas.  Bo  the 
director  hired  programmers  with  no  teaching  experience. 

These  programmers  tended  to  be  too  creative  by  writing 
lessons  that  did  not  conform  to  the  directives  of  the  lesson 
designers.  The  resulting  lessons  frequently  missed  the 
teaching  objectives  and/or  the  intended  level  of  difficulty. 

Case  3C1.3  - Graduate  student  programmers:  scheduling 
bottlenecks 

Une  curriculum  development  group  hired  graduate  stu- 
dents as  programmers.  Sometimes  this  proved  to  be  a bottle- 
neck in  lesson  production.  They  scheduled  their  programming 
duties  around  their  courses  and  studying  needs,  which  meant 
they  often  planned  to  program  at  night  or  on  weekends. 
However,  at  those  times  the  system  was  sometimes  down. 
Furthermore,  senior  staff  were  not  available  to  serve  as 
consultants.  The  students  who  were  on  assistantships  were 
entitled  to  all  university  vacations.  This  presented  a 
particularly  big  problem  when  the  vacation  was  a month  long 
break  between  semesters. 

2.  Instructor  with  programmers 

Case  3C2.1  - Inadequate  guidance 

An  administrator  picked  particular  courses  in  his 
department  for  development  of  COE  materials.  He  asked  each 
instructor  who  was  a content  expert  to  be  in  charge  of  his 
own  subject  matter.  A programmer  was  assigned  to  each  pro- 
fessor to  carry  out  the  project.  The  lessons  were  variable 
in  quality.  If  the  professor  provided  guidance  in  instruc- 
tional design  or  if  the  coder  had  teaching  experience,  the 
lessons  were  good.  However,  some  professors  just  turned 
over  content  information,  and  programmers  wrote  ineffective 
lessons . 

Case  3C2.2  - Absentee  lesson  designers 

A professor  who  was  interested  in  using  CBE,  but  who 
did  not  have  time  to  become  acquainted  with  the  medium, 
hired  a programmer  to  develop  lessons  for  him.  In  the 
course  of  a year,  the  programmer  (a  graduate  student  in  the 
subject  area)  acquired  an  understanding  of  appropriate  uses 
of  the  medium  but  was  frustrated  in  attempts  to  change  the 
direction  of  the  curriculum  being  produced.  His  super- 
visor's only  contact  with  CEE  occurred  during  monthly 
visits  to  view  the  latest  materials.  Suggestions  for  design 
approaches  tended  both  to  underutilize  the  capabilities  of 
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the  medium  and  to  make  unrealistic  demands  on  the  prop, rammer 
by  insisting  on  specific  effects  that  were  very  difficult  to 
' program  (and  which  were  often  pedagogically  equivalent  to 

- effects  which  were  easily  programmed). 

' In  another  subject  area,  a professor  with  past  experi- 

ence in  design  of  individualized  instruction  initially 
hired  a programmer  because  he  did  not  want  to  "waste  his 
time  with  the  details."  This  professor  spent  a great  deal  of 
! time  reviewing  and  modifying  the  lessons  he  designed.  He 

found  it  very  frustrating  to  have  to  work  through  an  inter- 
^ mediary  (particulary  since  he  could  see  how  easily  most 

' alterations  were  performed).  Finally,  the  professor  decided 

that  it  was  much  more  efficient  for  him  to  learn  to  program 
directly  himself.  He  ultimately  produced  many  hours  of 
' effective  instruction  for  CBE. 

I 

3.  Independent  authors 
Case  3C3-1  - Success 

In  at  least  three  cases  new  authors  were  graduate 
students  who  had  teaching  experience  and  were  subject  matter 
experts.  Even  though  they  had  no  programming  experience, 
they  were  able  to  learn  programming  and  to  write  useable 
lessons.  Part  of  their  success  may  have  been  due  to  the 
fact  that  their  early  affiliation  with  the  project  was 
similar  to  an  apprenticeship.  They  had  the  consultative 
support  of  a project  director  with  expertise  in  all  aspects 
of  authoring.  Furthermore  he  understood  that  they  needed 
some  time  to  learn  by  trial  and  error,  particularly  in  the 
early  phases. 

Case  3C3.2  - Shift  to  hiring  programmers 

One  group  began  by  having  each  author  design  as  well  as 
program  his  own  lessons.  The  co-leaders  found  that  they 
were  unable  to  maintain  a perspective  of  the  higher  level  of 
organization  of  the  curriculum.  So  they  decided  to  central- 
ize lesson  design  and  make  it  the  responsibility  of  just  two 
people.  Programmers  were  hired  to  carry  out  the  details. 

The  result  was  more  efficient  lesson  production  and 
generally  better  products. 

Case  3C3.3  - Shift  to  team 

At  two  sites  each  author  was  wholly  responsible  for  a 
lesson.  This  responsibility  consisted  of  planning,  writing, 
and  coding  the  lesson.  The  result  was  that  lessons  varied 
greatly  as  to  instructional  approach,  physical  characteris- 
tics, and  quality.  Also  since  the  authors'  experience  and 
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talent  varied,  some  depended  on  their  colleagues  for  extra 
, help.  Thus,  for  example,  one  of  the  authors  found  himself 

burdened  not  only  with  writing  the  lessons  assigned  to  him 
but  also  with  helping  colleagues  who  were  having  programming  | 

difficulties.  In  one  instance,  when  a senior  author  left  i 

unexpectedly,  the  project  foundered  because  nobody  was  ; 

available  to  help  the  less-experienced  authors.  Both  pro-  i 

jects  shifted  to  team  approaches  in  order  to  make  better  use  j 

of  the  available  talent.  i 

\ 1 

Team  approaches 

Case  3C4.1  - Uniform  procedure  provided 

At  one  site,  the  varied  quality  of  lessons  produced  by 
independent  authors  led  to  the  decision  to  establish  a uni- 
form procedure  of  lesson  development.  This  included  not 
only  tasks  that  were  necessary  for  lesson  development  in 
general,  such  as  writing  objectives,  criterion  tests,  and 
peer  lesson  review,  but  also  procedures  for  creating 
instructional  strategies  and  performing  formative  evalua- 
tion. 

This  decision  allowed  for  and  was  followed  by  a 
formal  division  of  labor.  It  resulted  in  more  rapid  lesson 
production,  and  lessons  that  were  relatively  uniform  with 
respect  to  strategies  and  quality. 

Case  3C^.2  - Standards  prespecified 

At  one  military  site  it  was  felt  that  the  staff  avail- 
able for  authoring  did  not  have  all  of  the  necesary  subject 
matter  expertise  for  developing  lessons.  Previous  experi- 
ence at  that  site,  using  independent  authoring,  had  resulted 
in  a wide  range  of  lesson  quality.  Some  of  the  lessons  were 
unnecessarily  elaborate  for  meeting  the  limited  objectives 
of  the  program.  The  decision  was  m.ade  to  organize  three- 
member  teams,  each  consisting  of  a content  expert,  an 
instructional  designer,  and  a CBF  expert.  The  administra- 
tors also  specified  standard  strategies  and  techniques,  and 
a lean  approach  to  lesson  design.  They  allowed  no  new 
teaching  strategies  and  no  use  of  new  hardware.  As  a 
result,  adequate  lessons  were  produced  in  the  allotted  time. 

Case  3C4.3  - Preliminary  guideline  and  analysis  provided 

At  one  site  a system  for  lesson  development  was  devised 
by  one  group  of  personnel.  Authors  in  the  CEE  project  were 
required  to  follow  this  process,  although  they  were  not 
involved  in  creating  the  plan.  It  was  anticipated  that  CBE 
lesson  development  could  be  speeded  up  because  the  initial 
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guidelines,  analysis,  and  objectives  had  already  been 
prepared  for  them.  The  creativity  of  the  CBE  authors  was 
limited  by  the  contraints  set  by  the  systems  developers. 

The  CBE  authors  did  have  the  advantage  that  they  did  not 
have  to  start  from  scratch  in  designing  lessons.  They 
found,  however,  that  the  components  of  the  process  were 
inadequate  for  preparation  of  CBE  materials.  As  a result, 
more  detailed  outlines  of  the  lesson  content  were  furnished 
to  the  authors.  In  addition,  subject  matter  specialists 
were  brought  in  to  lend  their  expertise. 

Case  3C4.4  - Group  brainstorming 

One  group  decided  to  formulate  initial  versions  of 
lessons  by  group  "brainstorming  sessions."  The  attempt 
failed.  It  was  difficult  to  get  authors  to  implement 
session  suggestions.  Authors  felt  this  system  jeopardized 
their  autonomy. 


D.  Lesson  Design 

Of  the  many  factors  in  lesson  design  that  require 
decisions,  three  are  represented  in  the  cases  cited  below: 

(1)  the  use  of  standardized  vs.  free-form  lesson  designs; 

(2)  the  use  of  minimum  standards  and  guidelines;  (3)  the 
degree  of  pre-planning  needed  before  beginning  on-line 
design . 

The  goals  of  the  project  related  closely  to  the  deci- 
sion about  standardization.  When  the  goal  was  to  teach 
minimum  basic  skills,  (often  found  in  a military  environ- 
ment), the  objective  was  to  train  the  students  to  a just- 
adequate  level  of  proficiency.  Such  lessons  put  more 
emphasis  on  score  than  on  helping  a student  reach  a very 
high  proficiency  (Klecka,  1977).  Such  lessons  often  used 
standardized  formats  and  strategies  to  achieve  their  goals. 

On  the  other  hand,  varied  techniques  constituted  an 
appropriate  approach  where  one  of  the  objectives  was  to  try 
to  determine  which  strategies  were  most  effective  in  terms 
of  student  performance  and  acceptance,  or  when  the  objective 
was  to  provide  breadth  of  experience.  Authors  of  lessons 
with  these  goals  discarded  standard  approaches  in  order  to 
teach  students  to  the  fullest  extent.  Compared  to  lessons 
with  standardized  formats,  these  lessons  varied  widely  in 
quality,  from  truly  excellent  to  essentially  useless. 

In  order  to  expedite  student  interaction  with  peri- 
pheral equipment,  data  collection,  and  lesson  debugging,  one 
project  set  minimum  standards  for  all  lessons.  In  at  least 
two  projects,  decisions  were  made  to  standardize  the  pro- 
gramming: certain  variables  were  set  aside  to  hold 

information  that  was  necessary  for  overall  curriculum 
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management.  V/hen  one  director  of  a large  project  failed  to 
make  such  plans,  the  results  were  chaotic. 

The  traditional  systems  approach  to  lesson  design 
demands  complete  prespecification  of  goals  and  methods. 

Some  authors  of  PLATO  lessons  ignored  the  sytems  method  and 
used  an  artistic  approach.  They  created  lessons  much  as  an 
artist  creates  a picture.  llo  hard  data  are  available  to 
evaluate  the  effectiveness  and  production  time  required  for 
lessons  done  this  way.  Many  experimental  independent 
authors,  who  began  with  the  systems  approach,  cast  this 
method  aside  in  favor  of  an  evolutionary  method.  They  often 
found  that  when  they  translated  their  plans  to  the  terminal, 
many  details  had  to  be  changed,  and  the  careful  planning  had 
been  a waste  of  time.  They  preferred,  instead,  to  pre-plan 
only  the  objectives  and  content  of  the  lesson.  Then  they 
designed  the  lesson  on-line  and  revised  as  necessary  as  the 
lesson  evolved. 

1.  Varied  techniques  vs.  prespecified  strategies 
Case  3D1.1  - Varied,  good  results 

One  director  of  a university  project  decided  to  try  out 
a variety  of  pedagogical  styles.  His  goal  was  to  try  to 
determine  which  aspects  of  lesson  design  were  most  important 
with  respect  to  student  acceptability  and  performance.  He 
was  unable  to  find  any  differences.  The  two  things  that  did 
affect  lesson  quality  were  the  availability  of  "help"s 
(supplementary  explanations  that  could  be  obtained  on 
request)  and  good  answer  judging.  That  is,  the  student 
wanted  to  be  judged  right  when  his  answer  was  correct.  He 
did  not  want  to  be  a "mind  reader"  and  try  to  figure  out 
what  form  of  the  answer  was  acceptable. 

Case  3D1.2  - Varied,  poor  results 

In  two  different  situations  the  decision  was  made  to 
allow  authors  complete  autonomy  in  choice  of  instructional 
strategies  and  presentation  techniques.  Although  well 
versed  in  the  subject  matter,  the  authors  were  relative 
novices  in  the  areas  of  TUTOR  programming  and  instructional 
design.  In  one  of  these  two  cases,  goals  were  not  clearly 
specified.  The  resulting  lessons  in  both  cases  varied 
widely  in  quality  as  well  as  in  style. 

2.  Standards  and  guidelines 
Case  3D2.1  - Standards 

In  an  early  phase  of  a large  curriculum  development 
project,  a considerable  amount  of  time  was  spent  designing 
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individual  lessons  and  refininp,  them  rather  than  tryinp  to 
do  the  whole  curriculum.  These  were  carefully  student- 
tested  for  specific  successes  and  failures.  In  this  way  a 
set  of  basic  standards  was  developed.  The  original  lessons, 
along  with  the  standards,  served  as  a model  for  subsequent 
lesson  development.  A considerable  number  of  the  paradigms 
were  later  used  repeatedly  with  different  content  matter. 

Case  3D2.2  - Standardization 

Several  groups  decided  to  reserve  specific  variables  to 
hold  specific  information.  Later  when  decisions  were  made 
to  implement  changes  in  curriculum  management,  the  lessons 
themselves  did  not  have  to  be  revised.  The  director  of 
another  project  found  that  his  independent  authors  used 
different  terminal  function  keys  for  different  purposes. 

The  students  were  needlessly  confused.  He  decided  on  some 
standard  uses,  and  all  lessons  were  revised  to  meet  these 
requirements.  This  greatly  improved  the  ease  of  student 
interaction.  Later  he  prepared  standard  initial  displays 
which  authors  could  easily  attach  to  new  lessons. 

Case  3D2.3  - Ho  standards 

The  director  of  one  large  curriculum  project  made  no 
decision  about  lesson  standards  or  models.  Each  author 
wrote  lessons  independently  of  the  others  and  made  little 
attempt  to  coordinate  efforts  or  standardize  lessons.  The 
result  was  a fragmented  set  of  lessons  of  variable  quality. 
The  totality  of  the  lessons  met  no  particular  goals  or 
objectives  and  it  was  difficult  to  evaluate  their  effective- 
ness . 

Epilog.  The  director  decided  to  divide  the  authors  into 
groups,  each  of  which  was  responsible  for  specific  topics  in 
the  curriculum.  Subsequently,  each  group  developed  its  own 
goals,  procedures,  and  standards.  Each  group's  lessons  were 
a more  complete  and  cohesive  whole  and  could  be  evaluated. 

Case  3D2.4  - Pacing 

In  one  subject  area,  the  students  began  to  expect  a 
certain,  fast-moving  pace  of  interaction  with  the  terminal. 
If  a problem  was  difficult  and  required  more  than  3-5 
minutes  of  thinking  before  any  answer  could  be  arrived  at, 
the  student  was  likely  to  complain,  and  write  a note  on-line 
to  the  author  in  a file  provided  especially  for  student 
notes.  The  author  decided  to  provide  help  for  questions 
which  the  students  found  difficult.  As  a result,  there  were 
almost  no  complaints  about  problem  difficulty. 
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Case  3D2.5  - Incomplete  use  of  the  lean  approach 

One  project  adopted  both  a "lean  approach"  and  mastery 
learning  approach  to  lesson  design.  The  chief  curriculum 
designer  decided  to  teach  only  what  was  necessary,  since  the 
goal  was  to  produce  qualified  individuals  at  a minimum  cost. 

However,  rather  than  modify  the  lessons  where  student  usage 
showed  the  original  content  to  be  inadequate,  the  original 
lesson  was  retained  and  students  were  routed  through  it 

repeatedly  until  they  passed  the  performance  test.  As  a i 

result,  the  lessons  provided  little  branching  for  remedia-  | 

tion  help  and  little  corrective  feedback  information.  If  a j 

student  failed  the  criterion  test,  he  had  to  repeat  material  j 

he  had  already  done  successfully  in  order  to  get  to  parts  he  j 

had  failed.  This  was  abrasive  to  students  and  made  ineffi-  ; 

cient  use  of  their  time,  but  it  was  more  efficient  in  terms  t 

of  authoring  time.  The  author  did  not  have  to  create  addi-  ■' 

tional  material  for  help  units  or  program  branching  alterna- 
tives. This  lesson  design  philosophy  had  one  other  disad-  i 

vantage.  The  authors  did  not  explore  alternative  lesson  ; 

formats  which  might  have  taken  longer  to  develop  initially 
but  which  could  have  reduced  training  time  needed. 

3 . Procedures 

Case  3D3.1  - Experienced  author,  shift  from  systems  approach 

One  director  initially  used  a systems  approach  to 
lesson  design.  He  found  that  when  he  put  the  lesson 
on-line,  it  turned  out  quite  differently  than  he  had  antici- 
pated. For  example,  the  screen  display  was  too  crowded  and 
the  essential  idea  was  lost.  He  decided  to  do  only  a mini- 
mum of  pre-planning.  fie  only  planned  the  topics  that  would 
be  covered  and  the  objectives  of  the  lesson.  All  of  the 
rest  of  the  lesson  design  was  done  on-line.  This  proved  to 
be  most  efficient  because  the  author  could  look  at  the  small 
sections  immediately  and  revise  as  he  went  along.  Moreover, 
new  ideas  were  spontaneously  generated  in  the  process  of 
iteration.  It  should  be  pointed  out,  however,  that  this 
author  was  very  experienced  in  using  the  CDE  system  as  an 
instructional  medium  as  well  as  in  his  subject  matter  and  in 
teaching. 

Case  3D3.2  - Experienced  author,  shift  from  detailed 
planning 

Une  director-author  began  developing  lessons  by  writing 
down  all  details.  After  completing  2 or  3 lessons  (about 
45-minute  instructional  sequences),  he  realized  that  parts 
of  them  had  to  be  rewritten.  After  that  he  decided  not  to 
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preplan  large  sections,  but  to  design  small  sections  at  a 
time  and  proceed  as  in  Case  3D3.1  above. 

E.  Peripherals 

Peripheral  equipment  added  new  capabilities  to  PLATO 
terminals,  but  also  brought  complications.  It  was  not 
always  immediately  available  and  experienced  users  were  few. 
For  example,  touch  panels  arrived  slowly,  and  it  was  often 
the  case  that  a touch  panel  was  not  available  for  every 
PLATO  terminal.  The  directors  had  to  decide  whether  or  not 
to  program  lessons  to  include  the  touch  capability.  In 
three  cases,  the  authors  decided  not  to  use  touch  until  all 
of  the  equipment  was  available.  Since  delivery  was  very 
late,  most  lessons  did  not  include  touch.  The  director  of 
another  project  included  touch  in  the  lessons,  but  held  up 
student  trials  until  all  of  the  needed  panels  became 
available.  A successful  alternative  to  the  dilemma  was  the 
use  of  a touch  simulation  until  panels  were  available. 

In  many  subjects,  teaching  is  greatly  enhanced  by  the 
capability  of  superimposing  terminal-generated  characters  on 
slides.  In  order  to  implement  this  technique  m.icrofiche  had 
to  be  produced.  PLATO  production  was  not  a quick  or  easy 
process.  The  attempt  to  shortcut  this  process  led  to 
disaster . 

The  audio  device  was  still  in  a prototype  stage  when 
one  project  director  made  a commitment  to  use  it  exten- 
sively. Much  of  his  staff's  time  was  devoted  to  testing  it 
and  making  it  easily  available.  This  greatly  decreased  the 
time  the  staff  could  spend  on  lesson  development.  Other 
project  leaders  decided  not  to  use  audio  because  trial  les- 
sons demostrated  it  was  poor  quality  and  unreliable. 

1 . Touch  panels 

Case  3E1.1  - Not  used  until  available 

In  at  least  four  cases  touch  panels  arrived  slowly. 

Not  all  terminals  had  panels  when  the  authors  v/ere  planning 
and  developing  lessons.  If  they  decided  to  incorporate 
touch,  the  lessons  could  not  be  used  on  some  terminals.  If 
they  did  not  use  touch,  this  nice  feature  would  not  be  in 
the  lessons  even  when  panels  were  available  later.  At  three 
sites,  the  director  decided  not  to  use  touch  input  until  all 
panels  had  arrived.  Consequently  few  lessons  incorporated 
this  feature.  At  another  site,  the  touch  panel  was  included 
where  appropriate.  However,  there  were  no  student  trials  of 
any  lessons  until  touch  panels  were  available. 


Case  3K1.2  - Successful  alternative 

« 

In  at  least  three  projects,  programmers  were  unable  to 
"i  test  their  touch  programs  for  lack  of  available  equipment 

during  regular  working  hours.  Thus  they  had  to  work  during 
non-prime  time  when  systems  maintenance  resulted  in  frequent 
interruptions.  An  alternative  plan  used  by  one  group  was 
: : touch  simulation.  Touch  was  incorporated  where  appropriate 

and  lessons  could  be  used  even  without  touch  until  the 
; panels  arrived. 

2.  Microfiche 

' Case  3E2. 1 - Shortcuts  didn't  work 

Francis  (1976)  has  provided  a detailed  account  of  the 
difficulties  encountered  in  the  production  of  microfiche. 

One  decision  that  always  ended  in  disaster  was  the  attempt 
to  expedite  production  by  shortcuts  of  one  sort  or  another. 

3.  Audio 

Case  3E3.1  - Time  constraints 

The  director  of  a project  committed  himself  to  use  of 
audio  when  it  was  in  an  early  prototype  stage.  It  was  to  be 
an  essential  part  of  instruction.  In  order  to  insure 
a vailabl il ity  of  the  audio  equipment  in  sufficiently 
reliable  form,  much  of  the  group's  time  was  spent  testing 
hardware  and  providing  service  software  so  the  audio  could 
be  used  easily.  This  severely  limited  time  available  for 
{ needed  course  development  work  and  for  field  testing  of 

materials.  Late  delivery  of  hardware  also  limited  access  of 
students  to  the  material.  Total  productivity  for  the  group 
t in  terms  of  hours  of  instructional  material  per  hour  of  time 

I expended  was  over  2000hr/hr. 

[.  Audio  disks  could  not  be  reproduced  fast  enough  to  meet 

the  needs  of  all  users.  Lessons  could  not  be  used  without 
the  audio.  The  intended  population  was  dismayed.  Also, 
slow  reproduction  siphoned  off  staff  time  and  limited  the 
number  of  lessons  that  could  be  produced  at  a much  needed 
higher  cognitive  level. 

Case  3E3-2  - Conscious  decision  not  to  use  audio 

At  least  four  project  directors  felt  that  an  audio 
device  would  be  very  desirable  for  their  CEE  lessons.  In 
some  cases  their  students  were  poor  readers.  In  others, 
sound  such  as  a human  heartbeat,  was  an  essential  part  of 
instruction.  They  wrote  mini-lessons  to  test  out  the 
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quality  and  reliability  of  the  audio  device  of  the  CEE 
system.  They  found  the  sound  was  poor  quality,  partly  due 
to  excessive  background  noise.  The  audio  was  also  unreli- 
able: sometimes  messages  were  wrong  and  occasionally  they 
simply  did  not  occur.  Consequently  the  directors  decided 
not  to  develop  lessons  that  depended  on  that  audio  device. 

Case  3E3o3  - Dabbling 

In  several  instances,  authors  "dabbled"  with  peripheral 
equipment.  For  example,  they  included  microfiche  in  just 
one  place,  or  touch  responses  in  a few  displays.  They 
mistakenly  thought  they  were  only  supposed  to  use  these 
peripherals  when  absolutely  essential.  For  example,  they 
were  to  use  the  touch  only  in  instances  where  it  would  be 
very  difficult  to  have  the  student  make  his  response  by 
pressing  a key.  In  every  case,  the  results  were  bad.  The 
author  did  not  take  enough  time  to  understand  how  to  use  the 
peripheral  device  effectively.  Consequently,  the  affected 
parts  of  the  lesson  did  not  work  properly. 

F.  Implementation  and  integration  of  CBE 

It  was  not  the  case  that  PLATO  lessons  were 
automatically  integrated  into  existing  programs.  Charac- 
teristics of  successful  integration  were:  (1)  lessons  were 
easy  for  the  instructors  to  use;  (2)  information  about 
student  progress  was  provided;  (3)  instructors'  requests 
were  given  careful  attention  and  consideration  even  though 
they  could  not  always  be  met;  and  (4)  a proctor  was  avail- 
able in  the  classroom.  Integration  with  traditional 
instruction  was  facilitated  when  the  instructors  were 
involved  some  way. 

Major  difficulties  included  instructor  resistance  to 
CPE  and  resistance  to  using  somebody  else's  lessons.  Uni- 
versity students  resisted  doing  the  lessons  when  they  did 
not  consider  them  to  be  a regular  part  of  instruction,  like 
lectures  and  laboratories.  Some  techniques  that  helped 
alleviate  the  problem  were:  (1)  scheduling  PLATO  sessions  in 
the  time  table;  (2)  giving  extra  points  for  completing  les- 
sons; and  (3)  including  some  questions  from  PLATO  lessons 
on  the  hourly  tests.  Students  also  objected  when  the  ter- 
minals were  far  from  their  usual  classroom  buildings. 

In  some  institutions,  one  classroom  of  terminals  was 
available  for  many  classes  in  many  disciplines.  Scheduling 
problems  were  particularly  difficult  when  terminals  were 
assigned  for  evenly-distributed  use  by  each  class  but  the 
lessons  required  concentrated,  day-after-day  usage. 
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1.  bactors  that  contributed  to  successful  implementation 
Case  3F1.1  - !'ake  it  easy  for  the  instructor  to  use 

The  initial  plan  in  one  project  was  to  give  the 
teachers  control  over  sequencing  lessons.  The  hope  was  that 
this  would  encourage  them  to  integrate  the  CDE  lessons  with 
the  rest  of  their  teaching  in  this  subject.  However  this 
involved  a fair  amount  of  attention  to  detail  on  the  part  of 
the  teachers  and  was  not  easy  to  implement.  In  another  pro- 
ject, the  director  developed  a considerable  body  of  lessons 
for  his  discipline  and  was  eager  for  others  to  use  them, 
too.  He  made  it  easy  and  convenient  to  use  the  lessons, 
even  including  on-line  tests  (although  they  were  not  called 
tests).  The  more  convenient  he  made  it,  the  greater  was  the 
general  acceptance. 

Case  3F1.2  - Inform  the  instructors  about  student  progress 

In  some  cases,  CBE  terminals  were  part  of  the  class- 
room equipment.  Some  students  did  CBE  lessons  while  others 
were  engaged  in  different  activities.  The  instructor  was 
too  busy  to  observe  the  students  when  they  were  using  the 
CBE  lessons.  Yet  they  wanted  to  know  how  their  students 
were  doing.  Initially  the  project  director  decided  to  pro- 
vide on-line  information  about  how  each  student  was  pro- 
gressing, and  in  great  detail.  liowever,  there  was  too  much 
for  the  instructor  to  read  and  too  much  data  for  him  to  get 
a general  idea  of  how  the  students  were  doing.  The 
instructor  simply  did  not  use  the  detailed  information  when 
it  was  available  only  in  this  form.  In  this  project,  CBE 
lessons  were  just  one  aspect  of  classroom  instruction.  The 
project  director  decided  to  make  two  changes.  (1)  Data  made 
available  to  the  instructors  was  very  simple  and  referred 
only  to  key  lessons.  Usually  all  of  it  could  be  presented 
in  one  screen  display.  (2)  A printed  copy  of  the  data  was 
made  and  given  to  the  instructor.  The  result  was  that  they 
were  able  to  tell  very  quickly  how  well  the  students  were 
doing.  If  they  did  not  have  time  to  look  at  the  data 
on-line,  they  could  and  did  take  the  printed  copy  home  with 
them  and  look  at  it  at  their  convenience. 

Case  3F1.3  - Listen  to  instructjrs 

In  two  large  curriculum  projects  decisions  about  the 
lesson  content  were  made  at  the  development  center.  In  the 
early  phases,  instructors  were  not  overly  enthusiastic  about 
CBE.  Those  who  used  the  materials  frequently  made  requests 
for  different  or  additional  lessons  as  well  as  particular 
kinds  of  information.  The  authoring  staff  was  often  not 
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able  to  meet  most  of  these  requests.  However,  when  the 
staff  took  the  time  to  listen  to  the  instructors  and  talk  to 
* them,  the  instructors'  attitudes  improved. 

Case  3F1.M  - Lead  time  for  instructors 

Elementary  school  teachers  were  very  busy  during  the 
school  year  and  did  not  sign  on  to  the  terminal  very  often. 

. ■ Therefore  they  did  not  see  many  of  the  lessons  that  were 

developed  during  the  year.  Nor  did  they  gain  the  "comfort" 

1 or  feeling  of  ease  with  the  machine  that  comes  from  inter- 

' acting  with  it.  The  teacher's  attitudes  toward  CBE  instruc- 

tion improved  with  his  degree  of  comfort,  and  his  attitude 
; was  reflected  in  the  students.  When  possible,  time  was 

' provided  during  the  summer  for  teachers  to  review  the 

; ’ materials  and  experience  working  as  a student.  The  result 

was  that  they  were  more  at  ease  using  the  CBE  system  and 
more  likely  to  treat  the  lessons  as  an  integral  part  of  the 
class.  Students  then  picked  up  this  attitude  and  took  the 
CBE  lessons  seriously. 

Case  3F1.5  - Proctor  available  in  classroom 

One  project  provided  a classroom  proctor  and  also  an 
on-line  file  in  which  students  could  write  comments.  One  of 
the  factors  that  contributed  to  the  director's  decision  to 
keep  a proctor  on  duty  was  that  the  students  had  "better 
feelings"  when  a person  was  around.  In  fact,  the  tone  of 
the  comments  in  the  note  files  was  much  milder  when  they 
could  talk  about  their  comments  to  a proctor. 

Another  project  began  to  use  CBE  lessons  for  regular 
classes  before  the  lessons  were  carefully  revised.  The 
director  decided  to  have  proctors  in  the  classroom  to  help 
the  students  over  known  problem  spots  in  these  lessons. 

Thus  they  could  use  otherwise  unuseable  lessons  until 
authors  had  a chance  to  fix  them. 

Epilog . After  several  semesters  of  experience  and  consider- 
able lesson  revision,  they  decided  to  retain  a proctor  in 
the  classroom.  During  the  first  two  weeks  of  the  course 
this  was  necessary  to  take  care  of  various  beginning  of 
semester  problems.  After  that,  the  proctor  was  available  to 
help  students  with  related,  non-CBE  difficulties,  as  well  as 
to  write  notes  to  the  authors  about  pertinent  observations. 
He  also  encouraged  students  to  put  notes  in  the  comments 
file. 

Case  3F1.6  - Show  the  student  his  progress 

A curriculum  project  implemented  a mastery  learning 
strategy  in  one  section.  That  is,  the  student  had  to 
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achieve  a preset  criterion  in  order  to  move  on  to  the  next 
level  of  difficulty.  Initially,  the  student's  pro r, ness  was 
shown  only  to  the  instructor  and  not  to  the  student.  The 
students  thoup,ht  they  were  moved  from  one  lesson  to  another 
by  some  sort  of  map.ic.  Motivation  was  poor.  The  decision 
was  made  to  show  the  student  his  propress  and  to  explain  why 
he  was  movinp,  ahead  or  repeatinp  a section.  This  approach 
resulted  in  better  motivation,  and  increased  instructional 
progress . 

2.  Factors  for  successful  integration 
Case  3F2.1  - Printed  copies  of  lessons 

The  lesson  developers  in  one  project  were  eap,er  to  have 
the  teachers  integrate  PLATO  lessons  with  regular  classroom 
lessons.  In  order  to  encourage  this,  CRP  personnel  made 
worksheets  that  corresponded  to  the  PLATO  lessons,  often 
getting  electrostatic  prints  of  terminal  displays  and  repro- 
ducing them  for  the  entire  class.  This  had  many  benefits. 

It  enabled  the  teacher  to  observe  the  children  if  the  whole 
class  was  working  on  the  worksheets  at  the  same  time. 
Alternatively,  he  could  look  at  their  work  after  they  handed 
in  the  papers.  Otherwise,  he  might  not  have  much  opportu- 
nity to  observe  them  studying  the  lesson  because,  for  the 
most  part,  he  was  occupied  with  the  rest  of  the  class  when 
any  four  children  were  taking  their  turn  at  tne  CPF 
terminals.  Also  the  teacher  coul'l  take  the  printed  copies 
home  and  look  at  the  lessons  at  his  own  convenience.  The 
children  liked  the  idea  of  worksheets  related  to  their  PLATO 
instruction. 

Case  3F2.2  - Involve  instructors 

Early  in  the  development  of  each  of  three  curriculum 
projects,  lessons  were  written  at  the  development  center  and 
brought  to  the  participating  institution.  Students  took  the 
lessons  in  a special  classroom.  The  instructors  had  pre- 
vously  attended  some  orientation  meetings,  but  were  never- 
theless disinterested  in  the  CRE  program.  Later  in  the 
project,  the  decision  was  made  to  consult  the  instructors 
before  lessons  were  written  with  the  objective  of  coordi- 
nating them  with  standard  classroom  materials  and  methods. 
The  result  was  interest  and  cooperation  from  the  instruc- 
tors. The  students'  attitudes  changen  from  regarding  CRE  as 
an  "extra"  to  considering  it  an  integral  part  of  their 
studies. 
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Case  3FP.3  - Instructor  control 

One  project  director  organized  his  curriculum  in  a 
< tree-like  hierarchical  organization.  He  developed  a compu- 

- ter  router  which  automatically  made  all  decisions  about  the 

"optimal"  next  lesson  in  the  hierarchy.  Unfortunately,  the 
pedagogical  needs  of  the  students  were  different  in  each 
skill  area  and  the  generalized  decision  making  scheme  did 
not  meet  all  of  these  needs. 

Fpi log . The  decision  was  made  to  change  the  locus  of  con- 
^ trol.  Management  control  was  transferred  to  the  instruc- 

tors, who  set  up  a daily  list  of  lessons  for  each  student. 
The  results  were  as  good  as  or  better  than  those  under  the 
automated  system. 

' 3.  Implementation  problems 

Case  3F3.1  - Not  enough  terminals 

In  one  institution  many  different  courses  offered  CRF 
lessons,  but  there  were  not  enough  terminals.  The  decision 
was  made  to  allow  each  student  one  hour  per  week,  during 
class  time  if  possible.  In  one  discipline,  some  of  the 
lessons  were  intended  to  be  used  in  five  or  six  consecutive 
sessions.  Once  a week  was  not  effective  and  the  students 
could  not  get  the  impact  that  was  intended. 

Case  3F3.2  - Resistance  to  using  somebody  else's  lessons 

In  one  discipline,  CDF  lessons  were  prepared  on  topics 
that  were  part  of  the  standard  instruction  in  the  beginning 
courses.  These  courses  varied  slightly  according  to  whether 
they  were  intended  for  majors  or  as  a service  for  students 
from  other  departments.  The  department  head  made  the  deci- 
sion that  the  CBF  lessons  should  be  used  to  replace  lectures 
on  those  topics.  Instructors  showed  different  reactions. 

One  instructor  who  taught  "majors"  resisted  using  the  CBF 
lessons.  He  claimed  the  lessons  were  too  easy.  Another 
instructor  suggested  that  this  was  an  excuse,  and  the  real 
problem  was  that  the  individual  was  near  retirement  and  a 
victim  of  inertia.  An  instructor  who  taught  a service 
course  for  graduate  students  used  the  CBF  lessons  to  replace 
lectures.  He  complained  that  the  students  did  not  get 
enough  out  of  the  lessons  because  they  were  not  tied  to  the 
homework.  Instructors  who  taught  a large  service  course 
accepted  the  CDF  lessons  and  used  them  to  replace  one  of  the 
two  weekly  lectures  throughout  the  course.  They  considered 
the  lessons  quite  acceptable  because  there  was  no  decrement 
in  student  performance. 
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Case  3F3.3  - Resistance  to  CBE 

Une  group  of  individuals  in  a university  department 
developed  CBE  lessons  and  made  them  available  to  all  members 
of  the  department.  The  older  staff  who  were  near  retirement 
professed  an  interest,  but  never  found  the  time  to  look  at 
the  lessons.  They  refused  to  have  anything  to  do  with  the 
lessons  as  part  of  their  courses.  They  did  not  even  try  the 
lessons  or  observe  students  doing  them.  It  is  possible  that 
these  people  were  simply  not  interested  in  any  innovations, 
but  it  is  just  as  likely  that  they  were  wary  of  computers 
and  CBE  in  particular. 

Case  3E3.^  - Increasing  student  attendance 

In  several  projects,  some  students  did  not  consider  CBE 
lessons  to  be  a "regular"  part  of  the  course.  Others  found 
the  classroom  of  terminals  was  not  conveniently  located. 

The  instructors  found  too  many  students  were  not  trying  the 
lessons.  One  project  director  decided  to  give  two  extra 
points  toward  the  semester  grade  for  each  lesson  the  student 
completed.  Another  instructor  announced  that  some  of  the 
hour  exam  questions  would  be  taken  from  information  given  in 
the  CBE  lessons.  Both  methods  were  successful  in  increasing 
student  attendance. 

Case  3F3.B  - Administrative  dilemma,  games 

One  of  the  administrative  decisions  that  had  to  be  made 
was  whether  or  not  to  allow  game  playing  at  a site.  One 
problem,  seen  fairly  often,  had  to  do  with  "PLATO  drop- 
outs"--students  who  became  game  freaks  at  the  expense  of 
their  studies.  Another  consideration  was  the  adequacy  of 
resources  for  both  regular  lessons  and  game  playing.  In  at 
least  two  university  terminal  classrooms  the  site  director 
organized  a list  of  games  for  students  to  be  able  to  access. 
The  top  administrators  made  different  decisions  in  the  two 
cases . 

In  one  case,  the  site  director  set  up  a list  of  games 
to  make  the  PLATO  system  more  appealing.  The  project  direc- 
tor decided  that  this  was  a poor  policy  and  discontinued 
game  playing.  He  felt  it  would  encourage  students  to  think 
of  the  PLATO  system  as  a source  of  games  and  not  as  a ser- 
ious instructional  device.  He  also  felt  that  some  of  the 
students  who  were  being  attracted  to  the  classroom  were  not 
part  of  the  target  population.  Their  presence  was  certainly 
not  good  advertising  for  the  project  director  who  was  trying 
to  justify  the  advantages  of  CBE. 

In  another  case,  the  site  director  felt  that  games 
enabled  students  to  get  a refreshing  change  of  pace  from 


intensive  study.  Terminal  usage  was  completely  restricted 
so  problems  with  outsiders  never  arose.  The  top  administra- 
tors did  not  object  to  the  availability  of  games,  and  no 
serious  problems  were  reported. 
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Chapter  ^ 
EVALUATION 


Formative  evaluation,  that  is,  evaluation  during  lesson 
development,  early  became  a necessary  part  of  the  process  of 
lesson  development.  In  fact,  in  most  projects  continuous 
evaluation  was  an  integral  part  of  lesson  production,  imple- 
mentation and  maintenance.  Edwards  and  his  colleagues 
(1975)  have  suggested  that  at  least  four  kinds  of  changes 
take  place  as  a program  evolves.  They  argued  that  evalua- 
tion should  be  continuous  because  these  changes  are  contin- 
uous. The  changes  that  occur  in  the  development  of  a PLATO 
lesson  closely  parallel  those  that  they  enumerated.  (1)  The 
objectives  of  the  project  sometimes  changed,  as  from 
research  to  applications  orientation.  (2)  The  program 
changed  in  character,  such  as  from  being  self-contained  to 
being  an  integrated  part  of  a larger  whole.  (3)  The  program 
was  established  in  response  to  some  educational  or  societal 
circumstances,  and  those  changed.  For  example,  initial 
plans  for  an  innovative  curriculum  were  dropped  for  lack  of 
funds  or  changing  interests.  (4)  Knowledge  accumulated  and 
dictated  changes.  This  was  true  of  all  aspects  of  PLATO 
projects . 

It  was  important  for  each  PLATO  project  to  have  an 
evaluator  as  a staff  member  or  as  a consultant.  In  one 
incident,  where  there  was  no  one  person  in  charge  of  data 
keeping,  overlapping  responsibilities  and  an  inadvertent 
breakdown  in  communications  resulted  in  a considerable  loss 
of  data.  When  an  evaluator  was  not  consulted,  inappropriate 
statistical  techniques  were  employed  and  the  results  were 
difficult  to  interpret  and  generally  questionable. 

Planning  appropriate  formative  and  summative  evaluation 
during  the  initial  stages  of  project  development  was  essen- 
tial. By  making  such  plans,  one  project  was  able  to  evalu- 
ate the  effectiveness  of  what  was  accomplished  even  though 
they  did  not  meet  all  of  their  goals.  In  contrast,  another 
group  did  not  plan  for  evaluation  and  also  did  not  meet  its 
goals.  As  a result  they  could  not  even  evaluate  the  lessons 
they  did  complete. 

In  general,  data  were  used  not  only  to  assess  effec- 
tiveness but  also  to  revise  and  improve  lessons.  Formative 
evaluation  included  lesson  reviews,  student  trials,  and 
lesson  validation.  The  impact  of  lesson  reviews  on  revision 
depended  on  the  timing  of  the  review,  the  qualifications  of 
the  reviewer  and  the  nature  of  author-reviewer  relationship. 
End-of-lesson  reviews  by  the  outside  consulting  staff 
effected  few  substantive  revisions. 
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This  lack  of  acceptance  was  surmised  to  be  due  to  three 
psychological  factors  ( Call-Himwich , 1977).  (1)  Authors 

were  subjectively  involved  and  viewed  their  lessons  in  much 
the  same  way  an  artist  views  his  work.  (2)  By  the  time  an 
author  received  the  review  he  had  already  begun  work  on  a 
new  lesson  and  was  deeply  involved  in  it.  The  old  lesson 
was  "cold"  and  had  been  tucked  away  mentally  as  "finished." 
(3)  The  totality  of  so  many  suggested  revisions  may  have 
been  overwhelming. 

Reviews  made  during  lesson  development  increased  the 
probability  that  lesson  revisions  would  be  more  than  cosme- 
tic. Experienced  teachers  and  colleagues  who  were  also 
authors  were  usually  effective  reviewers.  Subject  matter 
experts  tended  to  look  only  at  content  accuracy.  It  was 
felt  that  university-oriented  reviewers  made  inappropriate 
suggestions  because  they  did  not  understand  the  environment 
or  needs  of  the  non-university  student.  Face-to-face  inter- 
active reviews  between  author  and  reviewer  seemed  to  be  more 
effective  than  written  reviews. 

Student  trials  were  important  for  b<  ' lesson  revision 
and  validation.  Collection  of  on-line  dava,  such  as  time 
and  record  of  responses,  enabled  the  author  to  revise  the 
lesson  to  fit  the  allotted  time  and  to  detect  problem  areas. 
It  was  necessary  for  authors  to  observe  student  trials  in 
order  to  note  trouble  spots  which  on-line  data  did  not 
reveal.  Reliable  information  was  most  likely  to  result  when 
the  students  who  tried  out  the  lessons  were  from  the 
intended  student  population.  One  project  made  specific 
plans  to  validate  lessons  and  this  validation  helped  smooth 
implementation.  Failure  of  other  groups  to  validate 
resulted  in  some  unuseable  lessons  and  many  frustrated 
students . 

At  least  four  project  directors  established  notes  files 
and  encouraged  students  to  make  on-line  comments  about  les- 
sons. One  director  said  they  were  valuable  but  three  direc- 
tors said  the  comments  provided  little  useable  information 
and  sometimes  they  were  in  bad  taste. 

In  at  least  four  projects,  the  student  could  request 
on-line  information  about  his  status  and  progress.  This  was 
particularly  motivating  and  saved  a great  deal  of  instruc- 
tors' time.  In  one  project  which  did  not  make  such  informa- 
tion available,  a major  complaint  from  the  students  was  that 
they  did  not  know  where  they  stood  in  the  course. 

Lesson  effectiveness  was  measured  in  different  ways. 
Some  projects  compared  test  scores  and/or  learning  time  with 
a control  group.  Others  did  pre-test,  post-test  comparisons 
to  determine  gains.  Most  projects  used  some  form  of  opinion 
questionnaire.  No  specific  cases  are  cited  below,  but  it 
was  generally  felt  that  these  questionnaires  provided  the 
project  director  with  useful  insights  about  the  students' 
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attitudes  toward  a particular  lesson  or  the  entire  CBE 
course . 

A.  Planning  Evaluation 
1 . Meed  person  in  charge  of  data 
Case  4A1.1  - Loss  of  data 

Evaluation  of  lesson  effectiveness  was  a major  goal  of 
one  project.  The  staff  did  not  include  an  evaluation  spe- 
cialist. One  objective  was  to  compare  CBE  lessons  to  non- 
CBE.  Students  in  both  CBE  and  control  groups  were  required 
to  take  a paper-and-pencil  quiz  at  the  end  of  each  topic  and 
to  meet  a specified  performance  criterion.  If  the  student 
passed,  he  received  a "pass”  rating  and  was  allowed  to 
continue  on.  If  he  failed,  he  received  a "fail"  rating  and 
had  to  repeat  the  lesson  and  quiz  cycle  again  until  he 
passed.  The  pass/fail  data  had  been  chosen  as  one  measure 
of  lesson  effectiveness.  The  other  measure  was  the  total 
time  it  took  the  student  to  successfully  complete  the 
lesson . 

Time  data  for  CBE  students  were  automatically  collected 
by  the  computer  and  transferred  to  permanent  storage  for 
later  use.  Times  for  control  students  were  recorded  by  the 
instructors  and  later  entered  into  a computer  file.  All  of 
the  pass/fail  data  was  to  have  been  entered  into  a computer 
file  but  the  computer  program  was  not  available  in  time. 
Therefore  the  course  personnel  recorded  each  student's  data 
as  he  progressed  and  later  collected  all  the  data.  They 
informed  the  CBE  staff  that  they  would  keep  it  indefinitely 
or  let  them  know  if  it  was  to  be  destroyed.  However,  when 
the  CEE  staff  requested  the  data,  they  found  that  some  of 
the  data  had  been  inadvertently  destroyed.  Unfortunately, 
not  all  of  the  original  data  was  recoverable.  This  loss 
could  have  been  prevented  if  the  responsibility  for  data 
keeping  had  been  assigned  to  a single  individual. 

Plans  for  statistical  analyses  of  data  were  made  with- 
out consulting  a specialist.  The  most  appropriate  statisti- 
cal tests  were  not  chosen,  and  the  results  were  not  as 
reliable  as  they  could  have  been. 

Case  4A1.2  - The  need  for  experienced  evaluator 

At  one  site,  there  was  no  experienced  evaluator  on  the 
staff.  flowever,  staff  members  patiently  collected  quite  a 
lot  of  data  before  they  started  analyzing  them.  A cursory 
glance  at  the  data  showed  that  they  had  been  crudely 
gathered  (by  non-project  staff).  After  improvements  to  the 
data  gathering,  it  was  found  that  non-CBE  instructors  felt 
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that  the  evaluation  data  being  gathered  could  be  unfavorable 
toward  slow  students.  So  the  staff  members  were  lenient 
toward  them  and  gave  them  "breaks"  to  make  them  look  better. 
Experienced  evaluators  would  have  begun  analyzing  data  imme- 
diately as  well  as  devising  tests  and  checks  for  the  valid- 
ity of  the  data. 

2.  Need  to  pre-plan  evaluation 

Case  4A2.1  - Rewards  of  planning 

One  director  selected  the  target  course  for  CBE  and  set 
up  the  hypotheses  he  would  test  even  before  authors  were 
trained.  As  a result,  project  development  moved  steadily 
toward  testing  the  specified  goals.  Although  some  of  his 
data  were  marginal  in  value,  the  director  was  able  to  pro- 
vide some  useful  evaluation. 

Case  4A2.2  - Shift  to  formal  evaluation  plans 

A site  had  the  responsibility  of  developing  instruc- 
tional materials  for  a large  portion  of  a course.  Either 
because  of  the  press  of  daily  events  or  because  of  a con- 
scious decision,  no  formal  formative  evaluation  program  for 
the  lessons  being  developed  was  instituted.  Some  lessons 
were  tried  out  in  trial  student  runs,  but  these  trial  runs 
were  haphazard  at  best.  Peer  lesson  reviews  were  recom- 
mended but  were  by  no  means  univeisal.  The  only  constant  in 
the  formative  evaluation  of  these  lessons  was  the  outside 
reviews  which  were  done  several  times  for  most  lessons. 

This  service  combined  with  site  peer  reviews  was  mistakenly 
presumed  to  be  as  good  as  thorough  student  testing  of  the 
lessons.  When  real  students  began  using  the  lessons,  sev- 
eral gross  errors  appeared  in  them,  causing  a great  loss  of 
confidence  on  the  part  of  the  students.  This  loss  of  confi- 
dence was  surely  a factor  in  the  eventual  failure  of  the 
project.  Second,  the  authors  were  forced  to  revise  lessons 
as  the  students  revealed  errors  in  them  rather  than  develop 
lessons  for  the  next  segment  of  instruction.  This  had  the 
effect  of  putting  the  staff  even  farther  behind  in  an 
already  heavy  schedule. 

Epilog . At  a later  time  when  the  project  was  again  reorgan- 
ized, a formal  plan  for  evaluation  was  written  and  the 
necessary  data  were  collected.  Though  not  perfect,  they 
seemed  adequate. 

Case  4A2.3  - Failure  to  plan  evaluation 

At  one  site,  no  evaluation  plans  were  written  ini- 
tially. Consequently,  as  the  project  progressed,  plans 
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had  to  be  written  and  rewritten  several  times  due  to  the 
lack  of  agreement  between  various  staff  members  who  were 
responsible.  Every  time  new  plans  were  developed,  a 
considerable  number  of  lessons  were  discarded,  because  they 
did  not  fit  the  newly  devised  plan. 

B.  Aspects  of  Formative  Evaluation 

1 . Lesson  reviews 

Case  4B1 . 1 - Timing 

In  one  instance,  a consulting  and  support  staff  wrote 
lesson  reviews  after  the  lessons  were  completed.  The 
reviews  were  submitted  in  writing  and  delivered  to  the 
author  one  to  two  months  after  the  lessons  were  completed. 
The  reviews  covered  all  aspects  of  the  lesson,  instructional 
and  programming  quality.  Although  the  authors  said  the 
reviews  were  helpful,  authors  made  only  minor  changes,  such 
as  correcting  misspellings.  Although  in  one  case  a review 
noted  a programming  error  which  would  halt  student  progress 
through  the  lesson,  the  author  took  no  action. 

Epilog.  Realizing  the  motivational  and  psychological  draw- 
[ backs  to  receiving  an  exhaustive  critique  long  after  the 

I lesson  was  "completed",  the  staff  experimented  with  various 

alternative  review  approaches.  They  finally  developed  a 
plan  by  which  lessons  were  critiqued  in-progress,  or  as  they 
were  actually  being  written.  Reviews  were  shorter,  fewer 
changes  were  suggested  at  one  time,  and  major  problems  could 
j be  dealt  with  before  they  became  chronic.  In  addition,  a 

' new  feature  allowed  reviewers  and  authors  to  go  through  les- 

1 sons  "together,"  each  seeing  the  same  screen  display  even 

' though  they  were  miles  apart.  Reviews  thus  became  more  an 

i;  exchange  of  ideas  than  the  monologues  they  had  once  been. 

Authors  responded  much  more  favorably  and  incorporated 
50%-75/5  of  all  suggested  changes. 

Case  4B1.2  - Reviewer  credibility 

I One  individual  from  the  supporting  institution  reviewed 

lessons  of  some  authors  before  he  had  met  them.  He  found 
' that  these  authors  were  not  as  receptive  to  suggestions  as 

t was  the  case  when  he  had  met  the  author  at  some  time  pre- 

1 vious  to  that  of  the  review.  Several  reviewers  found  that 

they  were  more  effective  when  they  reviewed  the  lesson  in 
the  author's  own  env  ironm.ent . The  reviewers  also  reported 
; that  they  were  more  " sympathetic"  with  the  author's  pro- 

blems after  they  made  a site  visit. 
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Case  4B1.3  - Reviewers  lack  experience  with  target  students 

In  three  cases,  the  consulting  staff  were  university 
people.  The  authors  rejected  the  consultants'  suggestions 
becase  they  felt  that  their  students  were  very  different 
from  university  students.  Consequently  the  criticisms  and 
recommendations  were  not  considered  applicable.  However  the 
reviewers  noted  that  many  of  their  suggestions  were  like 
those  stated  in  more  general  instructional  guidelines  at  the 
authors'  institutions.  The  reviewers  perceived  that  the 
authors  were  simply  using  these  arguments  as  an  excuse  for 
not  accepting  the  recommendations. 

Case  4B1.4  - Reviews  by  subject  matter  experts  or  peers 

The  organization  of  information  that  would  be  reason- 
able for  a content  expert  may  be  quite  different  from  that 
which  would  be  presented  to  a novice  (cf.  Glaser  & Resnick, 
1972).  Peer  reviews  were  important,  but  could  not  replace 
student  trials  for  information  value.  Peer  lesson  reviews 
were  sometimes  only  content  specific  if  peers  were  not 
actively  involved  in  CBE. 

Some  project  authors  did  not  like  to  make  the  lesson 
design  changes  suggested  by  subject  matter  and  other 
reviewers.  Changes  took  time  and  they  felt  that  if  the 
content  was  correct,  the  lesson  was  good  enough.  Issues  in 
lesson  design  were  regarded  as  a matter  of  author  whim  and 
unimportant  for  learning.  They  therefore  requested  that 
reviews  be  limited  to  content  only  and  disregarded  any 
design  comments  made.  As  a result  many  reviewers  looked  at 
only  one  lesson  and  refused  to  do  more. 

Case  4D1.5  - Teachers,  author-colleagues 

In  many  projects,  authors  who  worked  together  reviewed 
each  other's  lessons  in  an  informal  setting  during  develop- 
ment. Although  there  was  some  defensiveness  among  new 
authors,  experienced  authors  found  these  reviews  helpful  and 
sought  them  from  respected  colleagues.  In  the  elementary 
situation,  teachers  sometimes  added  perceptive  insights  for 
lesson  revision. 

2.  Student  trials 

Case  4B2.1  - Pre-plans  to  validate 

At  one  site  the  decision  was  made  during  initial 
planning  that  there  would  be  a specific  number  of  trials  per 
lesson  and  a given  number  of  students  per  trial.  The  result 
was  a rather  smooth  implementation.  At  one  site,  parallel 


lessons  were  written  for  CBE  and  for  non-CBE  media.  Plans 
were  made  to  validate  non-CBE  materials  before  transferring 
them  to  CBE. 

Case  4B2.2  - No  validation 

One  group  needed  a fixed  number  of  hours  of  lessons. 

The  project  director  did  not  make  any  plans  for  student 
trials  or  validation.  Students  authored  the  lessons  and  did 
not  have  the  time  or  experience  to  test  them  adequately  with 
other  students  before  classes  were  to  use  the  lessons.  The 
result  was  that  many  students  in  the  class  were  frustrated. 
They  could  not  always  complete  the  lesson  because  of  pro- 
gramming errors,  inadequate  answer  judging,  poor  explana- 
tions, etc. 

Case  4B2.3  - To  interpret  data 

Data  collection  is  an  important  part  of  CBE  lesson 
writing.  In  one  group,  developers  found  that  some  of  the 
data  seemed  unreasonable,  so  they  decided  to  monitor  the 
classroom  whenever  possible.  As  a result  they  found  explan- 
ations of  data  which  they  previously  could  not  understand  or 
interpret.  For  example,  one  student  did  the  work  for 
snother,  or  a long  time  delay  was  the  result  of  a conversa- 
tion with  others  around  h^m. 

Case  4B2.4  - Authors  didn't  observe  student  trials 

At  one  site,  authors  had  been  told  that  they  should 
observe  during  student  trials,  but  refrain  from  helping  the 
students  unless  drastic  errors  occurred.  They  were  also  in- 
informed  that  it  is  very  difficult  for  an  author  to  stand  by 
when  the  students  struggle  with  his  lesson.  Some  staff 
member  therefore  decided  to  bar  authors  during  trials!  The 
only  information  that  the  authors  received  was  second-hand 
and  abbreviated,  of  course.  Consequently  they  failed  to 
learn  quickly  from  student  trials. 

Case  4B2.5  - Inappropriate  students 

In  one  case  students  from  another  course  were  used. 
This  was  not  a required  course  for  them,  and  they  did  not 
feel  responsible  for  learning  the  content,  so  they  had  a 
casual  attitude  toward  the  lessons.  The  authors  assumed 
that  the  lessons  would  be  satisfactory  for  the  target  stu- 
dents and  did  not  revise  the  lessons  as  they  should  have. 
When  the  intended  students  took  the  lessons,  they  saw  them 
very  differently.  They  could  not  learn  the  material  readily 
and  were  very  upset. 
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Case  4B2.6  - Students'  comments 

in  one  project,  questionnaires  were  handed  out  to  the 
students  to  help  the  authors  evaluate  the  lessons.  Students 
checked  off  attitudes  on  a 5-point  scale.  The  authors  found 
that  in  some  cases,  a lesson  might  get  high  ratings  from 
students.  But  when  they  observed  the  students  in  the  class- 
room they  had  difficulties  in  a few  places.  The  director 
decided  to  add  a note  file,  and  urged  students  to  leave 
comments  about  specific  problems  they  encountered.  The 
authors  found  these  were  extremely  valuable  for  revising 
trouble  spots.  Lessons  could  thus  be  brought  to  a highly 
polished  form. 

Two  other  projects  decided  to  use  note  files  because 
they  were  interested  in  student  reactions.  Neither  one  of 
them  found  the  notes  particularly  useful  for  lesson  revi- 
sion. Some  comments  were  in  poor  taste.  The  director  of 
yet  another  project  decided  that  notes  would  be  useful. 

He  anticipated  that  if  many  notes  were  sent  from  a parti- 
cular place  in  the  lesson,  it  would  indicate  that  it  was  a 
point  of  high  frustration.  However,  the  notes  were  too  gen- 
eral to  be  of  value  in  revising  lessons.  They  were  fre- 
quently derisive  and  discussed  the  instructor  or  the  CBE 
system.  One  condition  common  to  several  courses  whose  note- 
files  became  personally  vitriolic,  rather  than  instruction- 
ally  oriented,  was  that  in  these  notefiles  students  were 
permitted  to  read  and  respond  to  other  students'  notes. 

Some  instructors  found  it  useful  to  provide  a separate  place 
for  comments  which  were  unrelated  to  the  lessons. 

3.  Shift  in  procedure 

Case  4B3.1  - Shortcut  evaluation  procedure 

In  three  different  situations,  the  author-director  went 
through  the  following  steps  as  part  of  formative  evaluation 
(with  revisions  as  necessary):  review  by  colleagues,  trial 
with  a few  students,  trial  with  small  classes,  and  finally 
actual  use  with  large  classes.  As  authors  became  more 
confident  in  their  skill  at  lesson  production,  they  decided 
that  the  middle  steps  were  not  providing  enough  additional 
information  relative  to  the  time  invested.  Eventually  they 
decided  to  have  a colleague  review  for  content  errors  and 
then  put  the  lesson  out  for  regular  class  use. 

The  above  sequence  seemed  to  be  a common  element  of  the 
development  of  design  skills  in  a specific  subject-matter 
area  for  a fixed  target  population.  It  was  frequently 
reported  by  experienced  CBE  authors  in  interviews.  The  main 
danger  was  that  the  person  assumed  that  skills  learned  in 
these  specific  circumstances  apply  universally.  For 
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example,  an  attempt  to  provide  types  of  "personalized" 
feedback  that  was  found  very  effective  for  young  children 
failed  completely  for  adult  students  who  reported  that  the 
feedback  wasted  their  time  and  was  "childish". 

C.  Summative  Data 

1 . Reports  to  students 

Case  4C1.1  - Report  progress 

One  university  professor  supplemented  his  regular 
course  with  CBE  lessons.  He  found  that  most  of  the  students 
who  made  appointments  to  see  him  just  wanted  to  know  how 
they  were  doing  in  the  course.  He  decided  to  add  an  on-line 
capability  that  enabled  each  student  to  see  his  own  records, 
how  he  had  done  on  all  lessons  and  tests,  and  how  he  com- 
pared to  the  rest  of  the  class.  The  result  was  a dramatic 
drop  in  the  number  of  students  who  came  to  see  him  during 
office  hours  and  increased  reports  of  satisfaction  with  the 
course . 

Case  4C1.2  - Failure  to  report  status 

One  project  developed  a new  curriculum  for  CBE  but  did 
not  include  any  method  of  reporting  to  the  students  how  far 
they  had  come  or  where  they  stood  in  the  course.  This  defi- 
ciency, not  knowing  where  they  were  at,  was  a major  com- 
plaint of  students  who  protested  against  being  in  the  CBE 
sections . 

2.  Measures  used  or  needed 

Case  4C2.1  - Separating  out  specific  CBE  effects 

Many  groups  used  final  performance  scores  to  compare 
CBE  with  control  students.  This  was  not  a satisfactory  mea- 
sure because  CBE  effects  were  confounded  with  other  effects. 
Consequently  some  evaluators  found  significant  differences, 
but  many  did  not.  Authors  in  one  group  decided  to  write 
their  own  criterion-referenced  tests  to  compare  CBE  instruc- 
tion with  non-CBE  instruction  on  a set  of  lessons  on  a spe- 
cific topic. 

The  results  showed  that  students  completed  almost  all 
of  the  lessons,  but  the  time  they  took  varied  considerably. 
There  were  no  performance  differences  between  CBE  and  non- 
CBE  students.  However,  the  authors  used  specific  item 
information  to  improve  the  lessons. 
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Case  HC2.2  - Evaluation  should  include  information  about 
student  levels 

One  group  collected  time  and  performance  data  on  low 
ability-level  students  during  student  trials  of  their  les- 
sons. The  same  lessons  were  then  tried  by  a different, 
higher  level  student  group.  Data  revealed  ceiling  effects; 
the  material  was  too  easy  for  the  second  group,  so  the 
authors  decided  to  upgrade  the  level  of  the  materials.  They 
tried  the  lessons  on  a third  student  population  and  failed 
to  achieve  successful  performance.  The  authors  concluded 
that  the  lessons  needed  revision  again  to  meet  the  needs  of 
this  group. 

Case  4C2.3  - Record  of  time  spent  in  lesson 

Many  projects  kept  a record  of  time  students  spent  in  a 
lesson.  It  is  not  clear  in  which  cases  it  was  the  result  of 
a conscious  decision,  and  in  which  the  time  was  kept  because 
it  was  easy  to  obtain.  The  uses  of  the  time  data  varied. 
Time  spent  in  lesson  provided  useful  indications  of  the  need 
for  revisions.  If  students  needed  more  than  the  allotted 
time  to  complete  a lesson,  the  author  surmised  it  was  too 
hard  and/or  too  long.  If  a student  scored  poorly  on  a 
post-test  but  had  spent  considerably  less  time  than  average 
in  the  lesson,  the  author  assumed  that  the  failure  was  not 
the  quality  of  the  lesson,  but  the  failure  of  the  student  to 
complete  it. 

3.  Post-tests 

Case  4C3.1  - Burden  of  evaluation 

In  one  case,  lessons  were  written  at  a curriculum 
development  center.  Instructors  at  the  cooperating  insti- 
tution refused  to  give  post-tests  claiming  it  took  too  much 
extra  time. 

Instructors  at  another  institution  gave  post-tests 
before  students  had  completed  the  CBE  lessons.  Others 
waited  until  it  was  very  late  in  the  semester  and  students 
were  too  busy  to  take  them.  Result:  about  20^  of  the  stu- 
dents took  the  post-test  and  this  was  an  inadequate  basis 
for  evaluation. 

In  other  cases,  instructors  or  the  institution  were 
paid  for  the  work  entailed  in  administering  extra  tests. 

4.  Need  to  monitor  data  collection 

Case  4C4.1  - The  need  to  monitor  data  collection 

At  the  request  of  an  external  evaluator,  data  were  kept 
on  the  time  students  spent  in  lessons  in  one  project.  The 
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evaluators  intended  to  collect  data  for  a specific  class 
rather  than  for  all  of  the  students  who  did  the  lessons. 
Moreover,  they  intended  to  delete  the  data  of  those  students 
who  dropped  out  of  the  class  during  the  year  because  such 
subjects  would  not  be  availble  for  posttests.  The  data 
gathering  program  was  written  and  data  were  collected.  No 
individual  was  assigned  the  responsibility  for  it.  After 
the  project  was  completed,  it  was  discovered  that  the 
program  was  collecting  data  on  all  students,  and  not  limited 
to  those  in  the  study.  Furthermore,  the  program  for 
dropping  students  was  deleting  students  from  the  end  of  the 
list  rather  than  those  who  had  dropped  out.  If  someone  had 
been  specifically  assigned  to  monitor  the  data  collection, 
these  errors  could  have  been  corrected  early  in  the  project. 

Case  4C4.2  - Inappropriate  tests 

An  outside  evaluator  devised  test  items  to  measure 
the  achievement  of  students  taking  a set  of  C3F  lessons. 
Authors  reviewed  the  items  and  found  them  to  be  insensitive 
to  the  objectives  of  the  lessons.  Authors  felt  that  the 
outsiders  were,  in  fact,  lacking  familiarity  with  the  CBE 
system  and  its  impact  on  the  students. 

In  another  project,  the  CBE  instructor  agreed  to  use 
the  final  test  written  by  the  traditional  classroom 
instructor,  for  a comparison  of  media.  Forms  of  the  same 
test  had  been  used  for  several  years,  and  the  CBE  instructor 
assumed  that  prior  tests  defined  the  desired  course 
objectives  (which  the  CBE  material  had  been  designed  to 
teach).  However,  the  traditional  classroom  instructor  radi- 
cally altered  both  test  and  objectives  at  the  last  minute. 


Chapter  5 
MAINTENANCE 
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Lessons  need  to  be  maintained  and  classrooms  should  be 
monitored  even  after  lessons  have  been  successfully  used  for 
many  classes.  A number  of  reasons  support  this  contention. 

1 (1)  Information  may  need  to  be  updated.  (2)  New  program 

errors  may  occur.  (3)  Better  ways  of  presenting  the  mate- 
I rial  may  become  apparent.  (4)  On-line  data  don't  tell 

' everything  about  what  students  were  doing  in  the  classroom. 

And,  finally,  (5)  characteristics  of  the  target  population 
I may  change.  Critical  incidents  are  not  available  to  support 

' every  one  of  these  reasons,  but  experience  is  accumulating 

' to  suggest  that  they  are  quite  realistic. 

A.  Classroom  Management 

1 . Proctors 

Case  5A1.1  - Smoother  implementation 

Students  in  one  course  were  sometimes  irritated  at 
particular  points  in  lessons  and  wrote  very  negative  com- 
ments. The  director  decided  to  have  a proctor  available 
during  class  time.  As  a result,  students  tended  to  discuss 
problems  with  the  proctor.  Whenever  such  a proctor  was 
available,  the  typed  comments  of  students  were  consistently 
less  negative.  In  addition,  proctors  could  keep  authors 
posted  on  new  errors  which  had  previously  gone  undiscovered. 

Case  5A1.2  - Prevent  problems 

One  professor  was  short  of  funds,  so  he  did  not  hire  a 
proctor  for  his  CBE  classroom.  As  a result,  some  students 
signed  in  under  more  than  one  name.  Essentially  they  took 
away  the  learning  privileges  of  others.  Many  terminals 
needed  adjustment  for  using  microfiche.  Students  needed 
instruction  on  how  to  do  this.  The  professor  decided  to 
hire  a proctor,  at  least  for  the  first  few  weeks  of  the 
semester . 

Case  5A1.3  - Need  for  skillful  classroom  proctor 

In  one  case,  an  unqualified  person  was  in  charge  of  a 
classroom  of  CBE  terminals.  This  single  classroom  was 
intended  to  be  available  to  students  from  many  courses. 
Unfortunately,  the  classroom  was  overscheduled.  At  the 
beginning  of  the  semester,  the  number  of  students  exceeded 
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the  number  of  terminals.  Instead  of  explaining  the  problem 
to  the  students  and  attempting  to  make  some  other  arrange- 
ments, the  classroom  manager  was  rude  and  antagonistic.  As 
a result,  many  students  simply  refused  to  return  to  use  the 
terminals,  even  when  the  scheduling  problem  was  solved. 

Case  5A1.^  - Shifting  role  of  proctor 

One  department  decided,  on  the  advice  of  an  evaluator, 
to  have  a proctor  in  the  terminal  classroom  on  a regular 
basis.  When  lessons  were  first  used  for  regular  classes, 
some  of  the  lessons  were  known  to  have  problems,  such  as 
inadequate  answer  judging  at  specific  points.  The  proctors 
were  able  to  help  students  to  get  through  trouble  spots  and 
complete  the  lessons. 

After  lesson  revisions  and  several  semesters  of  experi- 
ence, these  problem  spots  were  ironed  out  and  the  proctors’ 
role  after  the  first  few  weeks  became  one  of  helping  stu- 
dents with  off-line  problems.  Other  roles  of  proctors  were 
to  write  problems,  to  send  notes  to  lesson  authors,  and  to 
encourage  students  to  write  comments. 
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Chapter  6 
SUMMARY 


Four  major  factors  that  helped  PLATO  projects  succeed 
were:  attention  to  developing  good  relations  with  instruc- 

tors and  administrators,  clear  goal  specification,  early 
plans  for  lesson  validation  and  evaluation,  and  plans  for 
contingencies  or  alterations  when  circumstances  necessitated 
J deviations  from  initial  plans. 

Staffing  was  particularly  difficult  in  the  early  years, 
because  nobody  knew  which  related  qualifications  were  most 
important  for  authors.  Almost  nobody  had  had  experience  in 
CBE.  It  turned  out  that  successful  authors  were  people 
whose  qualifications  eventually  included  teaching,  instruc- 
tional design  knowledge,  content  expertise,  and  knowledge  of 
the  TUTOR  language.  In  the  early  years,  new  authors  had  to 
learn  TUTOR  without  printed  handbooks  or  references,  because 
the  system  was  changing  too  rapidly  to  keep  printed  material 
up-to-date.  They  were  therefore  particularly  in  need  of 
psychological  support  from  a "master"  author  or  consultant 
to  serve  as  a model,  to  help  them  over  specific  problems  as 
they  arose,  and  to  guide  them  as  to  which  subset  of  the 
TUTOR  language  to  learn  first. 

There  was  no  single  best  model  of  staff  organization 
for  PLATO  lesson  development.  Some  groups  worked  as  inde- 
pendent authors  and  others  were  organized  as  teams  of  var- 
ious sizes.  An  advantage  of  the  independent  author  over  the 
teams  was  that  the  author  did  not  have  the  problem  of  trying 
to  communicate  his  ideas  to  another  person  for  implementa- 
tion; he  could  execute  the  lesson  exactly  as  he  chose.  He 
could  create  and  revise  as  he  developed  the  lesson  and  was 
not  constrained  by  the  necessity  of  prespecification  of 
details  for  somebody  else.  This  worked  best  for  authors  who 
were  experienced  teachers  and  who  became  proficient  pro- 
grammers. A disadvantage  of  independent  authoring  was  the 
variable  quality  of  lessons  that  resulted  in  some  cases. 
Authors  became  very  ego-involved  and  resisted  making  changes 
to  improve  lessons.  The  team  approach  did  not  engender  such 
ego  involvement.  Since  not  all  teachers  or  content  experts 
were  proficient  programmers,  the  team  approach  enabled  them 
to  concentrate  on  content  and  instructional  matters  and 
leave  the  coding  to  others.  The  most  successful  teams  were 
those  in  which  each  member  of  the  team  knew  something  about 
the  other  members'  domains.  A drawback  to  the  team  approach 
was  that  in  cases  where  a team  was  put  together  without 
clear  leadership,  internal  fighting  and  power  struggles 
often  occurred  with  resultant  loss  of  productivity. 
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Formative  evaluation  was  a necessary  part  of  lesson 
development  and  served  as  a basis  for  lesson  revision. 
Failure  to  plan  for  student  trials  often  resulted  in  gross 
programming  errors,  unuseable  lessons,  and  frustrated  stu- 
dents. Each  kind  of  evaluation  (lesson  reviews,  student 
trials,  on-line  data  collection)  provided  different  kinds  of 
information,  and  one  could  not  serve  as  a substitute  for  the 
other.  Lesson  validation  and  data  on  effectiveness  could 
only  be  obtained  when  summative  evaluation  had  been  pre- 
planned and  necessary  data  were  collected  and  monitored. 

Integration  with  other  instruction  required  special 
attention  to  involvement  of  instructors  in  a meaningful  way. 
Classroom  implementation  required  constant  monitoring  for 
overall  management  as  well  as  lesson  maintenance. 


GLOSSARY 


author:  an  individual  who  developed  and  executed  all 

aspects  of  a CBE  lesson:  content,  instructional  design, 
display  formats,  programming. 

CBE:  computer-based  education;  instruction  delivered  by 

computer . 

CERL:  Computer-based  Education  Research  Laboratory,  head- 

quarters of  the  University  of  Illinois  PLATO  system. 

director:  the  person  who  was  responsible  for  a CBE  project 

or  site. 

electronic  page  turner:  a CBE  lesson  which  is  essentially 
like  a textbook.  ‘ It  is  a succession  of  displays  which  pro- 
vide little  or  no  interaction  between  the  student  and  the 
computer . 

external  source:  an  institution  where  lessons  were  devel- 
oped, separate  from  the  school  which  enrolled  the  target 
population . 

instructor:  individual  who  teaches  traditional  material  and 

also  (possibly)  uses  CBE  lessons.  An  instructor  might  also 
be  an  author  of  CBE  lessons. 

on-site  consultant:  an  individual  who  was  at  the  same  phys- 
ical location  as  the  author  and  provided  help  to  overcome 
programming  problems. 

signon:  an  identification  given  to  a person  which  enables 

him  to  work  at  the  CBE  terminal. 

site:  a project  at  a given  geographical  location  or  a pro- 

ject developing  curriculum  for  a particular  subject. 

TUTOR:  the  special  programming  language  of  the  PLATO 

system . 
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